This commit is contained in:
@@ -3,41 +3,12 @@
|
||||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||||
|
||||
|
||||
todo: look at proper settings for rendering in docker on linux and try to make it resilient to crashing even if set too low or too little resources
|
||||
IDEAS:
|
||||
check guidelines for running in docker from link below
|
||||
check command line params for best fit
|
||||
see about re-using same or whatever
|
||||
check cpu, memory usage upon render request, reject immediately if the system is overburdened
|
||||
check for running chromium processes or some way to determine if in the middle of the last render for someone
|
||||
### SMALL QUICK ITEMS #####
|
||||
|
||||
Try a method to zap all chromium processes as a test to see if they *can* be killed whil stuck this way
|
||||
To test use the "custom date time format helpers" with 5k records, that reliably freezes everything even when others all seem to run ok
|
||||
(this in itself is curious but whatever)
|
||||
todo: Why is the delete button pushed to be exposed all the time, surely it's not that commonly used?
|
||||
|
||||
What I'm seeing is on a long render it returns a 504 gateway timeout to the client but it's still churning away in the background
|
||||
5k records and "custom date time format helpers" report will take about 3 minutes to render, maybe less and it will crash out for the client return 504 but then it will complete at some point
|
||||
Confirmed it *does* complete because I was able to download the 4.12mb 2144 page pdf manually from the server temp folder via filezilla once I saw the cpu go down again
|
||||
and chrome process stop
|
||||
90% memory is the max used in do graph panel, probably docker is not letting it take all or something isn't
|
||||
maybe swapping out is what's happening, peak memory usage seems to be half a gig or so but c
|
||||
97% cpu usage is max
|
||||
|
||||
I'm thinking it's a memory issue more than a cpu issue because the cpu is hardly pegged at all while it's rendering right up until it appears to run out of memory
|
||||
then it starts swapping and all hell breaks loose and the cpu pegs on the swap daemon
|
||||
Need to test by moving up to 2gb of memory witha resize but keep the single vcpu and see what's what
|
||||
also increase the timeout in nginx for the reverse proxy to wait
|
||||
|
||||
|
||||
|
||||
todo: take a look at front end cypress testing etc again, it's going to be the best way to assure quality going forward and indespensible for smoke / load testing
|
||||
|
||||
|
||||
todo: triage this stuff:
|
||||
|
||||
todo: Client is *NOT* automatically updating no matter what version numbers are set
|
||||
Need to look into this and fix it for certain as it will be a huge issue when beta / release generally
|
||||
Find the notes already about this, maybe there is a link about serviceworkers etc
|
||||
todo: LOGOUT UI button is still fucked on mobile
|
||||
needs a re-think, positioning it at the bottom is not working
|
||||
|
||||
todo: Default sample logos are fugly, make them smaller and cleaner and less ugly, maybe just a stylized text and faint background, classier looking
|
||||
|
||||
@@ -46,18 +17,25 @@ todo: Direct report view URL doesn't work if already logged in due to code only
|
||||
https://test.helloayanova.com/widgets/499
|
||||
localhost:8080/viewreport?oid=20000&rid=9
|
||||
|
||||
todo: attachment control drag and drop should work on both file list and also attach side of things
|
||||
todo: wiki drag and drop image should automatically attach the image and do the linkup etc
|
||||
todo: wiki image helper, if select ok but haven't selected a attachment it errors out, ok should not be available until there is a valid image selected
|
||||
todo: attach files drag and drop helper should be expanded to include the selection tab, not just the list tab
|
||||
todo: .webp type not available to wiki image helper when attached
|
||||
this is because it's not identified as an image when it's a .webp extension
|
||||
add to image types list for MIME
|
||||
|
||||
todo: notification subscription form tags control looks like ordinary tags control might confuse people that they are tagging that subscription
|
||||
Maybe change the text to "Automatic filter tags" or "Only if tagged with"
|
||||
|
||||
|
||||
todo: take a look at front end cypress testing etc again, it's going to be the best way to assure quality going forward and indespensible for smoke / load testing
|
||||
See where it's at, is any of it usable?
|
||||
|
||||
todo: DataListFilter UI pretty shitty
|
||||
error on duplicate
|
||||
how to make a new filter?
|
||||
|
||||
|
||||
|
||||
todo: make our logo smaller or move it when a biz logo is displaying on the login form
|
||||
|
||||
@@ -66,6 +44,10 @@ todo: hide swagger logo and branding in api explorer
|
||||
todo: Email notification sb link to edit subscription?
|
||||
Might be handy to directly click and turn off a subscription from teh notification?
|
||||
|
||||
|
||||
|
||||
### MEDIUM ITEMS #######
|
||||
|
||||
todo: chunk-vendors has fa-brands in it which is fucking huge, what else is in there I don't need?
|
||||
eradicate fa-brands!
|
||||
Looks like I'm doing it wrong, I was using a generic NPM module when there is a Vue specific one that also has guidance to limit to just the icons I actually use:
|
||||
@@ -73,44 +55,6 @@ todo: chunk-vendors has fa-brands in it which is fucking huge, what else is in t
|
||||
This could be a big breaking change so need to do it carefully and all at once as it's own project
|
||||
Definitely need to do though, it will save a lot of size to just include the actual icons I'm using which is a handful
|
||||
|
||||
todo: Why is the delete button pushed to be exposed all the time, surely it's not that commonly used?
|
||||
todo: LOGOUT UI button is still fucked on mobile
|
||||
needs a re-think, positioning it at the bottom is not working
|
||||
|
||||
TODO: ON UPDATE TO NEW version
|
||||
NOTE: This might all be due to package.json needing to have it's version updated
|
||||
it might be pulling that into the serviceworker for pwa or consulting it
|
||||
Right now it doesn't always refresh, have to manually do so, it should reload and ditch the cache on update, not sure why it isn't to be honest
|
||||
Try a test of it here locally and with release build, check network traffic, is it in fact actually updating and it's just the login page that isn't or...?
|
||||
|
||||
have an url that opens automatically or a notification and link to one after a new version has been detected just
|
||||
like visual studio does in order to show what is new in this version
|
||||
maybe ideally it opens to a new document page "whats new in version x.xx"
|
||||
this way no need to go beyond the local server or hit our site unnecessarily
|
||||
|
||||
todo: attachment control drag and drop should work on both file list and also attach side of things
|
||||
todo: wiki drag and drop image should automatically attach the image and do the linkup etc
|
||||
todo: notification subscription form tags control looks like ordinary tags control might confuse people that they are tagging that subscription
|
||||
Maybe change the text to "Automatic filter tags" or "Only if tagged with"
|
||||
|
||||
|
||||
|
||||
todo: export from grid to xls, csv etc.
|
||||
I'm thinking perhaps also a secondary ability to make report templates that are actually intended for export to xls or whatever as an additional feature
|
||||
but the easiest thing for people would be a straight export of grid data from grid, perhaps using the reportdata system since it pulls in all the bits needed for full data rather than just id links
|
||||
https://www.npmjs.com/package/json-2-csv
|
||||
From extensions?
|
||||
(because it's a single or list thing)
|
||||
|
||||
DASHBOARD / KPI / BIZ METRICS
|
||||
todo: DASHBOARD
|
||||
- Joyce kpi / charts and graphs
|
||||
- these cases:
|
||||
https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/2024
|
||||
https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/1974
|
||||
- Printable, need to test render these same metrics graphs in reports
|
||||
|
||||
|
||||
|
||||
todo: decimal-control and currency-control:
|
||||
Fix currency and decimal input controls since after vue-currency update latest version get error ".$ci is undefined" when open a form with them on it
|
||||
@@ -122,86 +66,45 @@ todo: DASHBOARD
|
||||
Plan: figure out if can ditch it as a dependency and if not then figure out how to work with it now as it's all different
|
||||
|
||||
|
||||
=-=-=-=-
|
||||
REPORTING STUFF THAT IS FUTURE OR ON HOLD
|
||||
MOVE THIS TO A v.next case or into reporting SPEC doc for future reference if not a coding issue now
|
||||
TODO: NEW CLIENT UPDATE ISSUES
|
||||
NOTE: This might all be due to package.json needing to have it's version updated
|
||||
it might be pulling that into the serviceworker for pwa or consulting it
|
||||
NOPE, doesn't make any difference setting package.json version
|
||||
|
||||
Right now it doesn't always refresh, have to manually do so, it should reload and ditch the cache on update, not sure why it isn't to be honest
|
||||
Try a test of it here locally and with release build, check network traffic, is it in fact actually updating and it's just the login page that isn't or...?
|
||||
|
||||
have an url that opens automatically or a notification and link to one after a new version has been detected just
|
||||
like visual studio does in order to show what is new in this version
|
||||
maybe ideally it opens to a new document page "whats new in version x.xx"
|
||||
this way no need to go beyond the local server or hit our site unnecessarily
|
||||
|
||||
todo: Client is *NOT* automatically updating no matter what version numbers are set
|
||||
Need to look into this and fix it for certain as it will be a huge issue when beta / release generally
|
||||
Find the notes already about this, maybe there is a link about serviceworkers etc
|
||||
|
||||
|
||||
todo: bizrule for report scale value at server
|
||||
and maybe other shit as well
|
||||
|
||||
todo: Need a setting that warns people about printing too much data, i.e. "That's a lot of data, are you sure you want to render that, it will be slow" or something to that effect
|
||||
### BIG ITEMS ###
|
||||
|
||||
todo: export from grid to xls, csv etc.
|
||||
I'm thinking perhaps also a secondary ability to make report templates that are actually intended for export to xls or whatever as an additional feature
|
||||
but the easiest thing for people would be a straight export of grid data from grid, perhaps using the reportdata system since it pulls in all the bits needed for full data rather than just id links
|
||||
https://www.npmjs.com/package/json-2-csv
|
||||
From extensions?
|
||||
(because it's a single or list thing)
|
||||
|
||||
|
||||
todo: REPORT BACKEND - report delete throws server exception about db context re-use or something to that effect
|
||||
No exact steps top repro, but happens for Joyce with win64 release build if go in and edit same template a few times from widget list then attempt to delete it
|
||||
try to repro on debug build figure it out, could be a big issue if it's not specifically a report issue but wider biz object issue
|
||||
COULD NOT REPRO HERE
|
||||
|
||||
todo: (On hold pending testing) pdf options UI and passthrough
|
||||
OUTSTANDING
|
||||
Docs
|
||||
DOCS reference pages
|
||||
https://stackoverflow.com/questions/49943479/puppeteer-header-and-footertemplate-doesnt-work#49996999
|
||||
https://pptr.dev/#?product=Puppeteer&version=v5.3.0&show=api-pagepdfoptions
|
||||
todo: document about the troubleshooting section items here if applicable:
|
||||
https://jsreport.net/learn/chrome-pdf
|
||||
which may or may not apply in our case
|
||||
Testing
|
||||
|
||||
DONE
|
||||
basically need to be able to select every option and send it through
|
||||
options:
|
||||
http://www.puppeteersharp.com/api/PuppeteerSharp.PdfOptions.html
|
||||
http://www.puppeteersharp.com/api/PuppeteerSharp.Page.html#PuppeteerSharp_Page_PdfAsync_System_String_PuppeteerSharp_PdfOptions_
|
||||
https://pptr.dev/#?product=Puppeteer&version=v5.3.0&show=api-pagepdfoptions
|
||||
page numbers control
|
||||
Test this, it might do what we need as it has a template for pdf footer and page number is part of it
|
||||
http://www.puppeteersharp.com/api/PuppeteerSharp.PdfOptions.html
|
||||
look at jsreport what do they include in their pdf post processing parameters and capabilities
|
||||
need to add pdfkit or whatever it's called at the front.
|
||||
|
||||
todo: I have console logging capture code now in backend, but it's doing nothing really, just logs if exception thrown
|
||||
it might be handy if it returned the log value and any other diagnostic info with render return data when user is in designer
|
||||
i.e. have a further property Diagnostic bool and if set then returns diagnostic data
|
||||
and return property with pdf name would be in an object with additional properties for diagnosis etc
|
||||
But only if it proves helpful or necessary
|
||||
mainly this would allow a console log or error or trace to flow back to the user from the script being run at the server
|
||||
|
||||
//before getting into timeouts and shit make sure it's running as well as can be in docker
|
||||
todo: look at guidance for running puppeteer (js) on alpine docker here: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-on-alpine
|
||||
how jsreport is launching headless chrome, i.e. which settings and flags etc
|
||||
https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#puppeteerlaunchoptions
|
||||
https://jsreport.net/learn/chrome-pdf
|
||||
https://github.com/jsreport/jsreport-chrome-pdf/blob/master/lib/conversion.js
|
||||
(after looking at it, it's still a bit unclear, maybe not relevant as they seem to do a lot differently for that)
|
||||
DASHBOARD / KPI / BIZ METRICS
|
||||
todo: DASHBOARD
|
||||
- Joyce kpi / charts and graphs
|
||||
- these cases:
|
||||
https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/2024
|
||||
https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/1974
|
||||
- Printable, need to test render these same metrics graphs in reports
|
||||
|
||||
|
||||
todo: look over this: https://github.com/puppeteer/puppeteer/issues/1834
|
||||
|
||||
todo: REPORT BACKEND - more timeouts for report rendering, hard kill after 30 seconds tops but adjustable interval maybe?
|
||||
each process takes time in part, need to see which is taking which time the most and killable
|
||||
i.e. run a huge report and see which exact step takes which time for each
|
||||
we have the pre-render timing out maybe need more for each step where it's vulnerable to crashing / timing out
|
||||
|
||||
Devops droplet is overwhelmed by 10k records of widgets using the customfields example report
|
||||
symptom is super high cpu usage (100%) pegged and probably virtual memory usage as well
|
||||
timeout is kind of a hard core way to work around this issue, maybe instead it should be looking at excessive cpu and memory usage?
|
||||
The metrics don't catch it because it happens too quickly for the metrics lifecycle
|
||||
|
||||
I want it to be able to handle it gracefully without crashing
|
||||
Looks like jsreports has this issue too and they hard kill the process I think if necessary, but they actually re-use the same instance for reporting I think.
|
||||
If it's not an issue then no need to resolve at the moment
|
||||
|
||||
UPDATE: No way to timeout the pdf generation built into puppeteer sharp so far, maybe some workarounds
|
||||
I think the nuclear option would be to start timer once we know the process id of the chormium instance and that time automatically kills that process ID if still found around XX seconds later
|
||||
(or in the form of a clean up job maybe like the temp file deleter job)
|
||||
|
||||
todo: reporting load test
|
||||
test locally with 20k widgets, make it crash then determine what limits to set on it and properly return error when it's exceeded
|
||||
right now it just says something about puppeteer and "crash"
|
||||
NOT ABLE TO REPRO HERE
|
||||
=-=-=-=-=-
|
||||
|
||||
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
|
||||
@@@@@@@@@@@@@ ROADMAP STAGE 5 - FINALIZE ALL NON BIZ OBJECT SPECIFIC FUNCTIONALITY
|
||||
|
||||
Reference in New Issue
Block a user