admin/raven
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
629cc300e49a8f84c5b769f426d5386e7d3636f3
raven/devdocs/todo.txt
John Cardinal 629cc300e4
2020-01-27 17:54:51 +00:00

553 lines
28 KiB
Plaintext
Raw Blame History

# SERVER TODO (J.F.C. - Just fucking code it already)
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOiIxNTcxODU5OTU0IiwiZXhwIjoiMTU3MjQ2NDc1NCIsImlzcyI6ImF5YW5vdmEuY29tIiwiaWQiOiIxIiwibmFtZSI6IkF5YU5vdmEgQWRtaW5pc3RyYXRvciIsImF5YW5vdmEvcm9sZXMiOiIzMjc2NyJ9.5irEX7LcuY7GL-Y_pBXiBSKgJFECDBem8PXezaNRvy8
## IMMEDIATE ITEMS
TODO: Ensure scaleability by checking for performance issues now before replicating code (particularly in widget etc)
- As per this document https://docs.microsoft.com/en-us/aspnet/core/performance/performance-best-practices?view=aspnetcore-3.1
- For scaleability go back to async only for any db calls functions like creating widgets etc
TODO: REFACTOR GetNoLogAsync function is used in many places redundantly when the logging version could do the same thing but not log it with an optional bool switch so refactor that shit
TODO: REFACTOR biz objects have two creates, an async and sync one, WTF is that about? See if can make it just one async version.
- https://docs.microsoft.com/en-us/aspnet/core/performance/performance-best-practices?view=aspnetcore-3.1
TODO: Need route to gather all object role rights in a format useful to display in UI so that a biz manager can see at a glance the rights for different roles to objects
- This way it's dynamic and picked up from the code itself which is always the source of truth so no need to put in the manual
- Would likely want to display multiple ways: for a specific object or role or selected user maybe too in the user info form ("effective roles")
- Move this over to client once the backend supports it
UPDATE SPEC DOCS with new format and names etc or at least remove nonsense as necessary
UPDATE MANUAL API REGARDING LISTS
TODO: DataFilter how to distinguish between filtering on specific ID value or on value column
- Might need to add a filter on ID type of thing maybe?
- Wait and see...YAGNI scenario maybe
DO CLIENT STUFF NOW COME BACK TO THIS STUFF LATER
TODO: the license being logged breaks the format of the log file because it has line breaks etc, so instead, maybe log out as a single line (remove breaks)
- Or maybe don't log it at all, what's the point? People can just edit it, maybe the DBID is the only really useful thing so we can match it for support to the customer
- or just log the ID number or a more succint version
\
*** BEFORE NEXT UPDATE TO DEVOPS SERVER:::::::
TODO: 2019-06-07 10:47:57.8894|WARN|Microsoft.AspNetCore.Cors.Infrastructure.CorsService|The CORS protocol does not allow specifying a wildcard (any) origin and credentials at the same time.
Configure the policy by listing individual origins if credentials needs to be supported.
TODO: "Client" -> "Customer " all languages but english need translated whatever was translated for "Client" text to whatever is now that languages "Customer" text (did english only for dev purposes)
TODO: "Service workorders" now "workorders" do for non english locales, keys changed are: GlobalWorkorderNumberStartSeed, WorkorderServiceTemplate, GlobalUseInventoryDescription, GlobalUnitNameDisplayFormatsDescription, WorkorderServiceList
- Also a shit ton more todo in all locales including english - just search for Service workorder case insensitive to see a bunch (in english anyway)
TODO: DO I NOT HAVE AN ERASE DB ROUTE?
- wouldn't that make the below possible??
TODO: API RATE LIMITING / THROTTLING
- Return code 429
- Going to need rate limiting to save people from themselves
- Have this link from solutions.txt about this: https://github.com/stefanprodan/AspNetCoreRateLimit
- All the big api's have hourly limits on requests and also limits on total data throughput per hour
- (minute as well which actually makes more sense since it means they would only be down for a minute if they exceed it) too
- Find out what others do
- Twitter: https://developer.twitter.com/en/docs/basics/rate-limiting
- Zendesk = 100 requests per minute max
- https://developer.yammer.com/docs/rest-api-rate-limits
- Good developer info here: https://medium.com/clover-platform-blog/conquering-api-rate-limiting-dcac5552714d
- Would be nice to have a switch to set for if we host as maximum rates they pay for maybe
- On the other hand it's likely a can of worms and maybe it's best they pay for their own hosting and we just set it up or something?
- but then it's going to be hard to self serve them where it's all done easy peasy unless we host
- Maybe it's a cadillac problem that can wait?
- Maybe DigitalOcean has rate limits we can set for the entire server and just mention it as a hard cap to customers
TODO: Is there a log that records past jobs, how long can they be viewed?
TODO: Rename the link to AyaNova App on the server page to just "AyaNova"
TODO: NEEDS A THINK, SOME CHANGES REQUIRE DB ERASURE BEFORE STARTUP
- Make it fucking easier to test deploy and erase db and fetch key and all that shit
- How about one click to erase the db, fetch the trial key and seed the data???
- I would like to deploy, bring it up and then from a simple command from the api explorer trigger all the above
- OR, ding ding ding ding...FUCKING CODE THE CLIENT SIDE FOR OPS SHIT LIKE FETCHING KEY, ERASING DB ETC!!!!
- Fixup the docker process on devops, make the db fresh on each boot by making postgres not save anything to volume and remove itself on downing like with the dev setup
TODO: Two kinds of mass fetch records in RAVEN
- 1) Reporting records
- Need to filter, sort and will probably pull in many different tables so would be a set of dedicated objects for that purpose at server
- Not using name display format as it's a reporting responsibility how the raw data is displayed at that point
- 2) Selection records
- USED IN
- In forms in pick lists for selection
- In main grids also for selection and viewing and triggering reporting (but reporting uses different lists to feed to report generator)
- Need to filter sort and search
- Uses templated name display format
- Server needs to take into account display format
- ACTION required:
- Make sure the server supports this
TODO: locale cjkindex, no way to set this value currently
TODO: Licensing - make sure tech type use license works properly: a tech license is consumed if a user has **any one** of the four roles of TechFull, TechLimited, SubcontractorFull, SubcontractorLimited
### TODO FILTER ISSUE: SEE integration tests search for BUGBUG there are two tests failing in DataFilterFilteringLists tests that might be DST related
TODO: JobsBiz::Create -> should log an event in order to be able to see who created a job, but maybe not for automatic internal jobs
TODO: Built in automatic job available that will regenerate sample data on a timer for display purposes. Maybe need a board with a variety of built in automatic jobs to choose from to enable or not since there will likely be others.
TODO: UserOptions fleshed out with client needs for formats:
- //TODO: also need the other locale settings such as number and date formats etc to be added at server
store.commit("setLocale", {
decimalSeparator: ".",
currencySymbol: "$",
shortDate: "YYYY-MM-DD",
shortTime: "hh:mm:ss A",
shortDateAndTime: "YYYY-MM-DD hh:mm:ss A",
timeZoneOffset: res.data.timeZoneOffset
});
=-=-=-=-=-=-
Less urgent things...
TODO: Need client docs at server end just for client user
- For now I'm putting them into the regular docs but that will be easy enough to copy and paste out later
- I just don't want the hassle of setting up another build process for client docs and folders and shit right now
TODO: UNLOCALIZED TEXT
- Look for old locale text keys that were not translated, there are more than a few keys that were added later to v7 and never translated out of english.
- Some may never be used so be careful about this.
- Unused keys won't be taking up any bandwidth so maybe this is a non-issue but you never know if I might need something in future as well so...tricky, maybe this is a 2.0 thing
DEVELOPMENT STREAMLINING
- VIRTUAL DEV
- Look into dockerizing all development tools in such a way they can be pulled in to a new computer quickly and start working
- Chromebook development? (they are currently 289 bucks so that's a greatr saving)
- I envision installing docker to a computer then issuing a single command to pull in all the requirements and launch vscode, the db server you name it
- This would then mean it would be safe to have a computer stolen or travel could just pick up a new one and get to work quickly
DOCS / MANUAL
- Disable google fonts for manual / docs generator:
- first need to upgrade to latest mkdocs AND material theme for mkdocs
- https://www.mkdocs.org/#installation
- https://github.com/squidfunk/mkdocs-material
- https://squidfunk.github.io/mkdocs-material/compliance/
- Once docs no longer need google fonts can adjust the devops nginx config and remove the overrides for CORS and all that shit.
DEVOPS
- Right now all that really matters is being able to test it and I can do that so the rest can be bumped.
- Wipe the server, create new at v 18 (or try the upgrade first just for kicks)
- Upgrade in place process: https://www.digitalocean.com/community/tutorials/how-to-upgrade-to-ubuntu-18-04
- Make devops AyaNova barebones without docker?
- Look into pros and cons because ultimately we'll want to run a bunch, maybe on demand, how to do that automatically? (Cadillac problem)
- Actually, won't they just be separate droplets? So really what I want is a very efficient droplet that I can clone?
- But then people want to just sign up and start working, not rent a whole server etc, it's up to us where we put the droplets or apps, but they don't want to see that
- Several scenarios:
- User signs up and starts working immediately, all through a subdomain, i.e. https://abcelectric.ayanova.com
- User wants to host it themselves on their equipment (nothing for us to do but sell the license)
- User wants their own host and domain i.e. https://abcelectric.com
- User wants to run it all locally on their own computer, docker image? Do we really want a windows installer route or...?
- initially probably manually set up so really not an affect either way as will just need to dedicate a port to each one regardless if dockerized or not
- We do want fastest perf though and no aggravation or cross exposure
- WIKI PAGE
- Doesn't require docs support as is now changed to a standard file attachment
- SERVER landing page
- I know TTM and all that but the server landing page should look nicer. Maybe a logo and better layout or something.
- If the Database connection is lost while AyaNova is running there are a lot of errors triggered by the generator etc.
- It should more gracefully handle this by perhaps closing down the api and generator should respect a closed API
- Set the reason to "Lost databases connection"
=-=-=-
some shit I probably don't need anymore:
CHOPPY DAY WORK
+++++++++++++++
https://severalnines.com/blog/performance-cheat-sheet-postgresql
https://medium.com/@wesharehoodies/immensely-upgrade-your-development-environment-with-these-visual-studio-code-extensions-9cd790478530
Changes needed to routes??
- http://www.talkingdotnet.com/actionresult-t-asp-net-core-2-1/
Ensure all modern best practice security is properly enabled on helloayanova.com so testing is valid
- https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security#Deployment_best_practices
CODING
++++++
refs:
- https://vuetifyjs.com/en/components/navigation-drawers
- https://vuetifyjs.com/en/examples/layouts/googleContacts
- Read this: https://qiita.com/nickhall/items/d1043f3f9874c90b6f8e
- And this: https://logrocket.com/blog/an-imperative-guide-to-forms-in-vue-js-2/
// eslint-disable-next-line
NEXT UP / CURRENTLY WORKING ON:
=-=-=-
IMMEDIATE ITEMS:
================
*** CLIENT SHELL ***
Go through the TOOLS.txt file VUE section and install
ServiceWorker - learn it, know it, love it
- Make Pecklist work with a service worker for offline use!!!!
- Do it "manually" to learn about SW better
- Use the VUE CLI generated stuff as a base
Do some tutorials to get a feel, read all the following
VUE PWA
- https://naturaily.com/blog/pwa-vue-cli-3
- https://dev.to/developerayo_58/vue-pwa-tutorial-how-to-build-efficient-progressive-web-apps-using-vuejs-5fdo
UI LIBS
OTHER
- Think about spa features
- Webpack (how can it help, what parts to use)
- Read about Single File Components https://vuejs.org/v2/guide/single-file-components.html
- Skim and have at hand ES2015 guide: https://babeljs.io/docs/en/learn
Once I can make the client I need to get into that and make the shell and initial interface with enough stuff to do basic testing initially
- Make sure to see the vue.js stuff in tools and below and contemplate it fully before committing to it
- VUE was chosen some time ago and there are likely other things out now
- Look and layout, graphics, logo, anything that is shell only
- Menu system
- Help link
- search
- Login , logout
- License
- Security / rights
- See localized text / change locale
later
- Widget CRUD and lists
- All input controls and date localization etc etc
- LOAD TESTER see below...
************************************************************************************
FUTURE ITEMS:
=============
LOAD TESTER
-----------
After very initial client stuff but before getting too deep into anything, need a load tester that drives the client and makes hundreds of thousands of objects and really exercises the client
and backend. I want to know what happens under typical heavy usage simulating months to years of work in just a few minutes.
I'm guessing something that autodrives a browser or simulates one to access via client UI for full end to end testing.
- Should be configurable to simulate various loads:
- Smoke test (Quick bug test): excercises one of everything once only looking for fundamental problems
- Load / scale / assurance test (Slower longer test to look for scaling issues related to a huge, busy office): exercises typical daily load of very large office but configurable to run in simulated days, so
can run it as a simulated week with 7 or month with 30 or years by entering in the number of days to simulate.
Would be nice if it did month end type stuff or delayed then billed out etc.
Should report stats about timing etc from the client perspective, we already have metrics for the backend perspective.
I'd like to know immediately if something is weirdly slow, not just in error.
Should not erase data at end so we can then manually login and check things out with a full load of at scale data
MISC FUTURE ITEMS NOT SURE ABOUT THAT CAME UP DURING CODING / TESTING
---------------------------------------------------------------------
- Should server show uptime somewhere?
- Log route: Add a route to download *all* log files as a single zip archive which will make life much easier for analysis
### ALL ITEMS
- LOCALIZED TEXT
- Localized text keys would be nice if they are understandable as is for API direct users so you don't need to be in the client to understand
what's happening
- Need a locale indepedent locale so that server errors without a corresponding user are localized to default english
- Also maybe a locale can be chosen at the server for error messages since we'll go by code numbers anyway.
- Starting to get to the point where I'll need this, i.e. error messages and logs that are part of core ops but need to be displayed in the UI
- Need to go through the api and find all the plain text messages returned and convert to locale text keys
- Ensure every error message has an error number of one kind or another and that they are not conflicted and easy to sort out if coming from server or api or etc
- Need to suck out our paid for translations and convert them into new locale text format
- DataDump??
- REPORTING
- Huge and unknown at this point Oct. 2018
- SEARCH TEXT
- See spec docs
- TECH SUPPORT
- Investigate how I can look at a customers DB with RAVEN
- Data masking for dumps is a start
- special "tech support" dump with masked customer information??
- Customer has a "key" that they can see which customer is the substituted masked one so we don't know the customer name but they can reference it themselves
- What about live looking at data through some feature?
- What would I need to look at or what information would I need?
- Be able to run a query directly and view results??
- Be able to run a query provided
- As a fix might need to enable customer to run a provided query.
- Be able to view all the meta information about the postgres instance
- Collation, sort order, languages, anything the user can set that could fuck up RAVEN
- MODIFICATION / CHANGE LOG (see case 79)
- Visible ID number generator case 3544
- Notification / generator / event of interest stuff (case 3491) BIG ONE
- Need interface, code for triggering notifications in biz objects ITriggerable :)
- Would it be more efficient to just process all notifications into the modification log regardless of subscribers
- CHILD objects need to point to their parent and be readable in code for searching and for opening objects based on child object
- so all child objects need a typeandid of the immediate parent
- Not sure where to document this so putting it here for reference
- Required for opening a search result of a descendent that is not directly openable an
- Need parent AyaType as an ENUM ATTRIBUTE in the AyaType table for easy traversal
- Import V7
- Tags - any type that is moving to tag can be coded now
- TESTING
- Longevity test on the DO server I can have up and running see core-testing.txt doc
CLIENT
- WHEN HAVE CLIENT - Localization (see core-localization.md)
- Time zone stuff (case 1912 related)
- WHEN HAVE CLIENT - Layout / Form user setttings
- WHEN HAVE CLIENT - Default form filling settings handling (case 3485)
- WHEN HAVE CLIENT - Push notification to client
- PUSH / POLL notification: determine and implement a system that can send notifications to client for things like
- change of localized text (invalidate cache)
- Server shutting down (log out asap)
- business object notifications (new workorder, status change etc)
- WHEN HAVE CLIENT Report route for widget
- WHEN HAVE CLIENT - test my PickList, is it sufficient? (pageable, alpha pageable (A-D, E-G kind of thing?))
- Case 1692
- search by tags plus text plus maybe pageable or...???
- WHEN HAVE CLIENT - ACTION / UI WIDGETS case 3460, 1729 UI as a collection of widgets stuff
- When widget is completely done, go over it and see if anything can be made easier or better before proceeding
- Generate seed data for tags
- Better to do this when the above core items are done as it touches on them
- Time zone
- This is not specced anywhere, but here, not sure where to put it at the moment, hopefully by the time I get here I will know (global settings? User settings?)
- Do not rely on the server's time zone setting, for example a docker container will be utc even if the server hosting it is pacific time
- Instead, use UTC for everything and have configurable value for timezone offset
### NFR
- UPDATE SWASHBUCKLE / SWAGGER to support testing file upload if not too onerous
- https://github.com/domaindrivendev/Swashbuckle/issues/280
- http://www.talkingdotnet.com/how-to-upload-file-via-swagger-in-asp-net-core-web-api/
- IMPORT / EXPORT
- biz object should import from v7, make a dummy import for widgets from something I've already exported in v7 (units? something with at least a name)
- Make a route for import to upload an import file? Then it runs the import via the biz objects and the correct ordering?
- OPS Functionality?? (maybe just biz admin only since it's dealing with actual biz data)
- OPS can import ops related stuff??(notification email server settings etc)
- BACKUP and RESTORE and COPY automatically to storage offsite
- Backup and restore data (widget, users etc)
- Close AyaNova server, erase db(optionally?? maybe user wants to combine two separate db's), restore the data
- See Discourse, they have some kind of AWS thing
- Also maybe this is handy: http://www.talkingdotnet.com/webhooks-with-asp-net-core-dropbox-and-github/
- Download backup, upload backup file
- FTP automatically?
- Need api speed test route (to independently of any particular object know how fast the connection is, for choosing a host site and troubleshooting)
- Some kind of static test list that is perfectly reproducible on demand
- Maybe a set routine of items to generate and return but in a way to disambiguate between slow server and slow connection
- don't re-invent the wheel
- What to test:
- compute performance
- DB performance
- thoughput? Speed of network
- Need some way to know if AyaNova is taking longer than it should to process requests so it can be an alert of some kind
- research how to time api avg running total or something, graph it for ops
- keep data by class of operation or tag it somehow
- Don't want it to actually slow performance
- Maybe have a benchmark time for various ops gathered during debugging tests, then hard code in that benchmark and if it takes longer then it logs it
- http://www.neekgreen.com/2017/11/06/easy-way-measure-execution-time-aspnetcore-action-method/
- https://weblogs.asp.net/jeff/asp-net-core-middleware-to-measure-request-processing-time
- SSL / TLS
- Need to look into how to support this
- Look into how the 2.1 dotnet will work with ssl so I do something relatively compatible
- REPORTING
- NOTE TO SELF: Don't report off Biz objects, make report specific objects. Better to have a reportclient list object and a selection client list object and etc than just a single client list doing duty as a selection box filler and a reporting object
And also biz object interfaces ideas:
ITaggable, ICustomFields, ISearchable, IExportable,IBizAction, IReportable (with sub interfaces for paging, format, report name and biz object for single and list etc,report stuff), ILocaleFields?, Etc
Your welcome!🤘😎
- CLIENT / UI DEVELOPMENT
- CLIENT UI "WIDGETS" ("COMPONENTS")
- Have UI testing scripts for developing UI. Scenario and then I can manually walk through it and see how ui responds to iterate from rough skeletal UI.
- Make a script for top X scenarios in the work day if each role. That way can try early rough designs with neutral expectations and reiterate until adequate.
- This way I won't design out of my ass without good input to riff off of.
- Orient express is some good shit for the shabs!
- Client: Start initial front end vue.js shell
- Need way to shut down clients gracefully (added value in api return? Polling [can't recall what the decision was in polling])
- Implement unlicensed server mode in client
- Request trial key from client, server fetches and installs
- Implement trial mode in client
- Seed data, erase db etc
- AFTER APRIL 1st 2018 - Dotnet 2.1 changes I must look into:
- will be rtm this summer-ish
- some swagger and webapi affecting changes
- efcore group by and lazy loading thing, might be relevant, nto sure
- HTTPS by default
- Look into it, see if something will be so huge that I should use the beta now for dev.
BUNDLING
- ONCE there is any front end code worthwhile then - Automatic build process Bundling and minification
- Parcel is coming on strong and requires supposedly zero configuration: https://parceljs.org/getting_started.html
- READ THIS: https://docs.microsoft.com/en-us/aspnet/core/client-side/using-gulp
- THEN SET IT UP
- Need automatic file copy script or whatever to copy docs to wwwroot folder somewhere so it can be served by the ayanova server
- Also need to package front end stuff for deployment as well with versioning etc, not webpack but along those lines
- Think about hostname being included with license, maybe a requirement?
- Localhost only or domain?
- Or would it be too much hassle with non-domain sites
- Fail2ban? Will we need that kind of thing incorporated into AyaNova?
- See again how it works and then look into application level ideas for that or what is smart for hardening, throttling etc
- 2FA Two factor authentication
- How hard is this to support in AyaNova?
- What about apps like Authy?
- It might be important to enable this for ops and biz accounts? Or at least be an option?
- https://blog.trailofbits.com/2019/06/20/getting-2fa-right-in-2019/
- https://news.ycombinator.com/item?id=20232164
- LETS ENCRYPT
- https://www.humankode.com/ssl/how-to-set-up-free-ssl-certificates-from-lets-encrypt-using-docker-and-nginx
- https://weblog.west-wind.com/posts/2017/Sep/09/Configuring-LetsEncrypt-for-ASPNET-Core-and-IIS
- https://stackoverflow.com/questions/48272373/how-do-i-get-letsencrypt-working-in-asp-net-core-razor-pages
- NGINX: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04
- Review again if need NGINX in front of kestrel still and if so then go this route
- DO WE NEED TO BE ABLE TO admin db from within raven ops route even if can't connect to db?
- Don't want users to have to use a db admin tool for anything, so should have ability to do whatever is necessary from ops route with db
- REQUIRED OPS: See if db exists
DEPLOYMENT AND TESTING
- DOCKER As soon as viable make an automatic build to a docker image for testing and deployment
- https://docs.microsoft.com/en-us/dotnet/core/docker/building-net-docker-images
- remote server online testing
- Better product will come from running it as it will be used as early and often as possible
- Look into renting the cheapest server on linode or digital ocean for dev testing
- set it up to pull the latest from repo so it automatically updates (or a docker image maybe)
- possibly set up integration test that goes off the remote server
- WINDOWS Automatic installer for testing
- Need a windows test bed and regular testing on it to confirm multiplatform interoperability
- Maybe a windows installer or maybe a docker image
- Integration test that can be pointed at any location to run a series of tests
MANUAL
- Add how to use swagger UI and authentication
BOOTSTRAPPING AUTHENTICATION
- Manager account can only login from localhost? - HMM...think on it
- What if can only create new users if manager account is changed from default credentials?
- that way you start your setup with one account, change it and then it's safe to do remote work
- by default manager account is only one with rights to configure server or user accounts.
- Doesn't have any rights to business config, only server config and CRUD user accounts
- So at least one admin user needs to be created locallhy before it can be used remotely to set up users
- manager account cannot be changed in any way, so always has default password and login
- JWT token check must check if local when it's the manager account user id 1
- This is so a user can't copy the creds from browser and use them remotely
- Test that shit from host
- What if can't run a browser in host for some reason???
- need an override that does allow remote manager account
- devise a way to bootstrap with no user accounts and a way to reset back to that
MAKE MVP
- Has the following features:
Alpha-0
- installer for windows and docker container
- VUE.js Front end that supports at minimum a login / logout and empty shell
- Shows server and client versions (about)
- https://vuejs.github.io/vetur/
- swagger docs and way to view them via the api
- User manual docs
- See the tools.txt section search for vue
Alpha-1
- Can do some minimal config like seed data, erase db etc
- Ops interface showing status and can view log etc
etc
LONG TERM::::::
## MVP and iterate
Reference in New Issue View Git Blame Copy Permalink