diff --git a/ayanova/devdocs/todo.txt b/ayanova/devdocs/todo.txt index 40e3277f..f91783d9 100644 --- a/ayanova/devdocs/todo.txt +++ b/ayanova/devdocs/todo.txt @@ -35,16 +35,77 @@ Case 3999 but what implications at front and back?? -## Generating HUGE dataset to confirm this and asked Joyce if this is new to this build and was not an issue before or not. - It could be the server is optimising itself based on the work load and revising it's algorithms for best query plan to run based on load and index retrieval etc?? - test with huge data, note particularly if it's slow at first then speeds up as it's used - has anything change that migth add slowness aside from new version of postgres? +TIMEOUT ISSUES ON HUGE LISTS + appears to happen with customer list on getting count, not data query + todo: + compare timing within AyaNova server and in pgAdmin for same query + can a timing report be added into the return data? + so we can see at the client when diagnosing how long each bit actually took? + or log that if it's excessive beyond xx seconds?? + Get a query plan and examine what's happening + See if it's just a badly written query in the first place and all else is well before getting into the weeds of alternate counting etc + research faster ways to handle count queries for all records, see if can speed it up + possibly need an alternative count method if it gets too slow or a timeout of some kind + maybe a separate query in datalist for known potentially slow ones? + but at the end of the day maybe the query is just a bad one and can be improved + the lateral stuff could maybe be a "with" type query using a chain instead of all inside as a subquery which is undoubtedly slower + + make a "mega" level of data generation that can be put up on devops to replicate test here + or, probably more practical, just up the HUGE level to crazy proportions beyond what any of our customers now likely have i.e. millions of records + who cares if it takes all night to generate if it provides useful testing + This is very important, we need to be confident it can work and perform well with huge amounts of data, i.e. millions of workorders and customers -I have not made a case, but there is a noticeable lag when moving to different datalists..... what sort of time lag would be expected when a datalist has (for example) 10768 workorder records? -i.e. 4073 customers for the Customers datalist <- a count of 4 seconds -i.e. 10768 workorders for the Work Orders datalist <- a count of 5 seconds -(when I next go to the datalist during that same "up" time for the server, even whenlogged in as another user, almost instantaneous which i believe is expected because then already cached) +///////////////////////////////////// +SLOW QUERY WORK AREA + +Baseline: +2021-10-14 10:24:11.2302|INFO|AyaNova.Api.Controllers.DataListController|(debug) DataListFetcher:GetResponse COUNT query took 49859ms to execute: SELECT COUNT(*) FROM ACUSTOMER +LEFT JOIN AHEADOFFICE ON (ACUSTOMER.HEADOFFICEID = AHEADOFFICE.ID) +LEFT JOIN ACONTRACT ON (ACUSTOMER.CONTRACTID = ACONTRACT.ID) +LEFT JOIN LATERAL + (SELECT serial AS LASTWORKORDERSERIAL, + SERVICEDATE AS LASTWORKORDERSERVICEDATE, + AWORKORDER.ID AS LASTWORKORDERID + FROM AWORKORDER + LEFT JOIN AWORKORDERSTATUS ON AWORKORDER.LASTSTATUSID = AWORKORDERSTATUS.ID + WHERE AWORKORDERSTATUS.COMPLETED = TRUE + AND AWORKORDER.CUSTOMERID = ACUSTOMER.ID + ORDER BY AWORKORDER.ID DESC + LIMIT 1) AS LWO ON TRUE + +LATERAL join is *all* the slowness exactly. It's ok in the data query because the limit is really saving our asses, but I bet a report on that would suck. So the fix needs to be done for data even though it's not slow + +rewrite as a WITH chain type query?? +{WITH qr AS (SELECT asearchkey.atype, asearchkey.objectid, COUNT(*) FILTER (WHERE asearchdictionary.word = 'fish') AS st1 FROM asearchdictionary INNER JOIN asearchkey ON asearchdictionary.id = asearchkey.wordid GROUP BY asearchkey.objectid, asearchkey.atype) SELECT atype, objectid FROM qr WHERE st1 > 0 } +NOPE same problem, runs once per entire query so returns last + +as a function as originally considered: +CREATE OR REPLACE FUNCTION public.aycustomerlastclosedworkorder( + customerid bigint, +out lastid bigint, +out lastserial text, +out lastdate timestamp) + LANGUAGE 'plpgsql' +AS $$ +BEGIN + + SELECT serial, + SERVICEDATE, + AWORKORDER.ID + INTO lastserial,lastdate,lastid + FROM AWORKORDER + LEFT JOIN AWORKORDERSTATUS ON AWORKORDER.LASTSTATUSID = AWORKORDERSTATUS.ID + WHERE AWORKORDERSTATUS.COMPLETED = TRUE + AND AWORKORDER.CUSTOMERID = customerid + ORDER BY AWORKORDER.ID DESC + LIMIT 1 +END;$$; + + + + +////////////////////////////////////// security jwt tokens and expiration, can a user just keep working if they are set to inactive because their token hasn't expired?