ayanova/devdocs/todo.txt

@@ -35,16 +35,77 @@ Case 3999
 	but what implications at front and back??
 	
 
-## Generating HUGE dataset to confirm this and asked Joyce if this is new to this build and was not an issue before or not.
-	It could be the server is optimising itself based on the work load and revising it's algorithms for best query plan to run based on load and index retrieval etc??
-	test with huge data, note particularly if it's slow at first then speeds up as it's used
-	has anything change that migth add slowness aside from new version of postgres?
+TIMEOUT ISSUES ON HUGE LISTS
+	appears to happen with customer list on getting count, not data query
+	todo: 
+		compare timing within AyaNova server and in pgAdmin for same query
+		can a timing report be added into the return data?
+			so we can see at the client when diagnosing how long each bit actually took?
+			or log that if it's excessive beyond xx seconds??
+		Get a query plan and examine what's happening
+		See if it's just a badly written query in the first place and all else is well before getting into the weeds of alternate counting etc
+		research faster ways to handle count queries for all records, see if can speed it up		
+			possibly need an alternative count method if it gets too slow or a timeout of some kind
+			maybe a separate query in datalist for known potentially slow ones?
+			but at the end of the day maybe the query is just a bad one and can be improved
+				the lateral stuff could maybe be a "with" type query using a chain instead of all inside as a subquery which is undoubtedly slower
+		
+		make a "mega" level of data generation that can be put up on devops to replicate test here
+			or, probably more practical, just up the HUGE level to crazy proportions beyond what any of our customers now likely have i.e. millions of records
+			who cares if it takes all night to generate if it provides useful testing
+			This is very important, we need to be confident it can work and perform well with huge amounts of data, i.e. millions of workorders and customers
 
-I have not made a case, but there is a noticeable lag when moving to different datalists..... what sort of time lag would be expected when a datalist has (for example) 10768 workorder records?
-i.e. 4073 customers for the Customers datalist <- a count of 4 seconds
-i.e. 10768 workorders for the Work Orders datalist <- a count of 5 seconds
-(when I next go to the datalist during that same "up" time for the server, even whenlogged in as another user, almost instantaneous which i believe is expected because then already cached)
 
+/////////////////////////////////////
+SLOW QUERY WORK AREA
+
+Baseline:
+2021-10-14 10:24:11.2302|INFO|AyaNova.Api.Controllers.DataListController|(debug) DataListFetcher:GetResponse COUNT query took 49859ms to execute: SELECT COUNT(*) FROM ACUSTOMER
+LEFT JOIN AHEADOFFICE ON (ACUSTOMER.HEADOFFICEID = AHEADOFFICE.ID)
+LEFT JOIN ACONTRACT ON (ACUSTOMER.CONTRACTID = ACONTRACT.ID)
+LEFT JOIN LATERAL
+	(SELECT serial AS LASTWORKORDERSERIAL,
+			SERVICEDATE AS LASTWORKORDERSERVICEDATE,
+			AWORKORDER.ID AS LASTWORKORDERID
+		FROM AWORKORDER
+		LEFT JOIN AWORKORDERSTATUS ON AWORKORDER.LASTSTATUSID = AWORKORDERSTATUS.ID
+		WHERE AWORKORDERSTATUS.COMPLETED = TRUE
+			AND AWORKORDER.CUSTOMERID = ACUSTOMER.ID
+		ORDER BY AWORKORDER.ID DESC
+		LIMIT 1) AS LWO ON TRUE 
+
+LATERAL join is *all* the slowness exactly.  It's ok in the data query because the limit is really saving our asses, but I bet a report on that would suck.  So the fix needs to be done for data even though it's not slow
+
+rewrite as a WITH chain type query??
+{WITH qr AS (SELECT asearchkey.atype, asearchkey.objectid, COUNT(*) FILTER (WHERE asearchdictionary.word = 'fish') AS st1 FROM asearchdictionary INNER JOIN asearchkey ON asearchdictionary.id = asearchkey.wordid GROUP BY asearchkey.objectid, asearchkey.atype) SELECT atype, objectid FROM qr WHERE st1 > 0 }
+NOPE same problem, runs once per entire query so returns last
+
+as a function as originally considered:
+CREATE OR REPLACE FUNCTION public.aycustomerlastclosedworkorder(
+	customerid bigint,
+out lastid bigint,
+out lastserial text,
+out lastdate timestamp)
+    LANGUAGE 'plpgsql'
+AS $$
+BEGIN	
+	
+	SELECT serial,
+			SERVICEDATE,
+			AWORKORDER.ID
+		INTO lastserial,lastdate,lastid
+		FROM AWORKORDER
+		LEFT JOIN AWORKORDERSTATUS ON AWORKORDER.LASTSTATUSID = AWORKORDERSTATUS.ID
+		WHERE AWORKORDERSTATUS.COMPLETED = TRUE
+			AND AWORKORDER.CUSTOMERID = customerid
+		ORDER BY AWORKORDER.ID DESC
+		LIMIT 1		
+END;$$;
+
+
+
+
+//////////////////////////////////////
 
 
 	security jwt tokens and expiration, can a user just keep working if they are set to inactive because their token hasn't expired?