raven/devdocs/research.txt

# Research required
“Do the simplest thing that will work.”


# BACK END / ARCHITECTURE


## DOCKER FINDINGS
	- Need a shared docker network between all containers, I was assuming could just redirect to localhost but didn't realize docker network is internal to docker 
		- this helped https://stackoverflow.com/questions/39202964/how-to-reach-another-container-from-a-dockerised-nginx?rq=1
		- first created a defined external network sudo docker network create docker-network
			- HOWEVER - I think docker has a default network so this external network probably isn't necessary, ??? MORE RESEARCH REQUIRED HERE
		- Then in the docker compose files of all involved had to put at the bottom one network statement (none in the service sections above)
			- networks:
  				docker-network:
    				driver: bridge	
			
	- need to run Nginx in front of ayanova for easiest ssl cert handling etc
		- Very good reference here: https://gist.github.com/soheilhy/8b94347ff8336d971ad0
	- Nginx needs to see ayanova via the docker host name which is the container name
		- So I had to point it like this:
			proxy_pass http://ayanova:7575;
				- Where ayanova is the image name in the docker compose file for starting AyaNova server








## DEFAULT PORTS (unassigned) http://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.txt
	- Some available ports not specifically assigned that I like the look of:
	- 4048, 4144, 4194, 4195, 4196, 4198, 4317, 4318, 4319, 4332, 4337-4339 , 4363-4365, 4366, 4424, 4434-4440, 4748, 5076-5077, 5551-5552,5994-5998
	- 7575-7587, 8084-8085
	- 7575 (RAVEN IS USING THIS)
	- 5995?
	- 4424?
	- 8084?

## ?? LICENSING 
    - ??how to functionally license RAVEN 
    - ??how to protect the keys

## ?? How am I going to expose the api for developers
    - Only through the rest interface?
    - Will there be plugins at the backend code level in any of those layers?
    - OUR STUFF VS END USERS STUFF
        - What existing plugins did we make and what would they need from Raven to be replicated?
            - how will this affect api decisions?
        - What can end users be allowed to do and how and at what layer.


## ?? FUNCTIONAL REQUIREMENTS /FEATURES

Look at AyaNova existing code,
A) what worked well
B) what was repetitious and should be automated
C) what had ongoing problems

Will help greatly with design

## TAGS
	- Wait to choose implementation type until basic design is decided upon as it will affect method use internally
	- This is actually a big adn tricky topic, here are some resources found while researching:
	- http://tagging.pui.ch/post/37027745720/tags-database-schemas
	- https://stackoverflow.com/questions/172648/is-there-an-agreed-ideal-schema-for-tagging
	- https://dba.stackexchange.com/a/35795  <---this is what I'm leaning towards
	- Leaning towards a single tags table with word and id (and maybe other properties like color, last used etc)
		- Each taggable other table has a map of it's own to the tags, so i.e.[TAGS: id, text] [CUSTOMERTAGS: customerID, tagId], [WORKORDERTAGS: WOID, tagID]
		- This will require a bit of complexity to keep it pruned though, i.e. tags will get orphaned when no one is using them so a routine will periodically need to clean them out.
		- This may be a better way to handle full text indexing as well with a search table for each object table.  Kind of balloons the table count though, but who cares really, does that matter?
	- An alternative is exactly like how I do full text search in AyaNova now with a generic foreign key, upside is cleaner db structure, but downside is ensuring cleanup and maint.
		- If parent object is deleted must go in and remove any linked tags as well.
		- Upside is it's easily searchable for all incidents of a specific tag over all taggable types, but is this really necessary?  It can still be done with separate queries.
		

** SEPARATE FOR *MVP* PRIORITY ONE IS MUST HAVE BEFORE RELEASE 8.0 / PRIORITY 2 OR LOWER IS NICE TO HAVE **
- go over all rockfish AyaNova related cases and request for features, into the hopper.
    - Move stuff to v8 if going forward, otherwise drop it's priority to lowest in AyaNova 7.x and any related project cases
    - Move any items that just have to be done now for v7 into high priority AyaNova 7.x track.
    - ?? What are the commonalities of things required in the UI to come from the backend based on current AyaNova 
        - Name ID lists?    		
        - list factories
        - report factories
        - fetch by name / id
    - Email sending and processing




## ?? CONCURRENCY ISSUES 
    - In the ef core book the author mentions getting around a concurrency issue by making it a non issue by not even updating the same order but instead appending status changes to another table
    - https://livebook.manning.com/#!/book/entity-framework-core-in-action/chapter-8/v-7/171
    - "This design approach meant that I never updated or deleted any order data, which means that concurrent conflicts could not happen. It did make handling a customer change to an order a bit more complicated, but it meant that orders were safe from concurrent conflict issues."
    - Microsoft tutorial: https://docs.microsoft.com/en-us/aspnet/core/data/ef-mvc/concurrency
    - See what will be critical concurrency in RAVEN (inventory likely), see if can code around it so no concurrency can happen.
    - Plan on what is concurrency prone and how to deal with it.
    - Maybe split out the concurrent sensitive bits of a workorder (parts) to a different screen, i.e. show read only in workorder adn to update that specific section have to go in and do it like a wizard or something?

## ?? BACKDOOR / LOST PASSWORD
    - How to handle this, current method is horribly susceptable to nefarious purpose
    - Maybe requires "physical" access to the server (put a file up or something?)
	
## ?? AUTOMATED BACKGROUND PROCESSOR "GENERATOR"
	- What will replace "Generator" in RAVEN?
    

## FRONT END

- VISUAL DESIGN / UX
	- - https://www.ventureharbour.com/form-design-best-practices/
	- Make the form designs simple and beautiful, it will directly translate into sales
	- Field labels above fields, not beside them (https://research.googleblog.com/2014/07/simple-is-better-making-your-web-forms.html)
	- Single column vertical layout for complex forms is definitely best.  In wide mode though what to do?
	- It's ok to have two inputs on a line together if they are directly related so that they have one single label above them and minimal space between them
		- i.e. date and time ([day] [month] [year] [hour][minute] etc)
		- But really don't if possible because it's never better than a single field that just "figures out" the parts of the input.
	- Always indicate required fields, don't just error if they are not filled out.
	- Break big forms into sections with shaded headers or emboldened headers with larger fonts but shaded background is good even with no text to break apart.
	- Use single fields for phone numbers or postal codes and process and interpret it through code rather than forcing them to use multiple fields as this causes confusion
	- Messages to users that are important and long and required to convey need to *really* stand out to even be read.  In their own box with a different background or something alike.
	- Put validation errors to the right of the input field, not below it as users can see it better and it requires less cognitive load.
	- Multi page forms (like wizards) need to have progress indicators or people will get antsy and bail
	- Favor checkboxes or radio buttons over drop downs unless there are more than 6 or so options as they require less cognitive load
	- Use placeholders sparingly (light gray prompt inside input) only when there is ambiguity, don't use them for obvious stuff 
	- Always use a predictive search / autocomplete for any field that requires a selection from a large number of options
	- Selectable images are the most compelling engaging selection type for users 
		- I.E. like a row of radio buttons but a row of images to select from and the choice is the selection by clicking on the image.
		- People enjoy clicking on images.
		- Use this as much as possible wherever possible.  I.E. for things like
			- A trial user picking the company type they would like to trial data for
	- Input fields should be sized to reflect the amount of data required to be entered in them 
		- This helps the user understand what is required of them
		- ZIP code or house number should be much shorter than an address line for example
	- Do not rely on colour alone to communicate
		- 1 in 12 men have some degree of colour blindness
	- Ensure entire forms can be navigated using the tab key.
	- Test the form in low and bright light situations on a mobile device outdoors and desktop
	- Enable browser autofill with contextual tags 	
		- Not exactly sure what this means in practice but it sounds good
	- Use visual cues and icons, brains process visual cues way faster than text alone
	- Don't ask for a password twice, use a eyeball icon to reveal text instead like keepass does
	- Mobile should be at least 16px high text / desktop can be 14px
	
	
- FRONT END FRAMEWORK
	- https://stackshare.io/stacks  (interesting tool to see what other companies use in their stacks)
	- https://semantic-ui.com/
	- Vue.js  https://vuejs.org/v2/guide/comparison.html
		- JS IN HTML via directives (kind of like Angular)
		- State management library if required: https://github.com/vuejs/vuex
			- Similar to redux but different
		- Module for VS Code!! https://vuejs.github.io/vetur/
		
		- PROS:
			- GUIDE: https://vuejs.org/v2/guide/
			- Simpler to code than React "Vue has the most gentle learning curve of all js frameworks I have tried"
			- Single file components, all aspects of a component are in the same file (css, js, html)
			- UI LIBRARY: http://element.eleme.io/#/en-US
			- Everyone keeps claiming it's more productive
			- Comes with a databinding and MVC model built in
			- CLI project generator
			- Has official routing solution and state management solutions
			- They are starting to work on a native library like react native (not there yet though and maybe I don't care about that so much)
			- Getting things done over "purity"
				- http://pixeljets.com/blog/why-we-chose-vuejs-over-react/
			- Working with html forms is a breeze in Vue. This is where two-way binding shines.
			- Vue is much simpler than AngularJS, both in terms of API and design. Learning enough to build non-trivial applications typically takes less than a day, which is not true for AngularJS
		- CONS:
			- Less answers on stackoverflow than React (for example)
			- Vue on the other hand only supports IE9+ (not sure if this is a con or not)
			- Doesn't have a major corporation behind it like React or Angular
			- (2016)runtime errors in templates are still a weak point of Vue - exception stacktraces in a lot of times are not useful and are leading into Vue.js internal methods
	- React.js
		- HTML in JS 
		- PROS:
			- Very widely used, tons of resources 
			- Works with huge apps
			- Can be compiled to native apps (somehow, though Vue is working on it)
		- CONS:
			- Requires a good grasp of javascript
			- Harder to code
			- Requires a *TON* of add-on libraries as it only deals with the view part
			- "PURITY" over getting things DONE
			- Very nitpicky and fuckery prone just to do things the official way, slower to get business objectives accomplished
	- Ember
		- PROS:
			- Requires less knowledge of javascript
			- Comprehensive includes all bits
			- Suited to small teams
		- CONS:
	- Angular
		- PROS:
			- Google and Microsoft supported 
			- Supposed to be good for someone coming from C# (typescript)
		- CONS:
			- Requires a *lot* of learning to use
				- Typescript
				- Angular directives and ways of doing things which are peculiar to it
			- Possibly slower than others to render
	

- ARCHITECTURE / TECHNOLOGY STACK
	- Electron hosted desktop app like vs code or Slack:
	- What is the advantage of Electron hosted app vs just a plain html 5 app? (nothing)
	- https://slack.com/downloads/windows
	- https://blog.bridge.net/widgetoko-a-node-js-and-electron-application-written-in-c-1a2be480e4f9
	


- PERFORMANCE: Do not send one byte extra that is not needed at the client
    - Not even a single space, 
    - especially not extra libraries or unminified code or cases
    - ICONS and webfonts should have only what is needed, nothing more.
    - ??USING A CDN??
- VERSIONING STATIC RESOURCES
    - Use Gulp, process with hash *in filename* not as a query string (better supported by intermediate caching or proxy servers)
    - https://docs.microsoft.com/en-us/aspnet/core/client-side/using-gulp
    - https://code.visualstudio.com/docs/editor/tasks
    - https://hackernoon.com/how-to-automate-all-the-things-with-gulp-b21a3fc96885
    - https://github.com/sindresorhus/gulp-rev
    - https://github.com/jamesknelson/gulp-rev-replace
- FORM VALIDATION: 
    - https://developer.mozilla.org/en-US/docs/Learn/HTML/Forms/Form_validation
- ?? framework for complex UI required or plain old javascript? (imagine a workorder or PO)
    - Need a lot of shit on forms, maybe a framework is the way to go:
        - VALIDATION / DIRTY CHECK
        - AJAX FEEDBACK
        - ??

	
- ?? Funky graphs
- ?? Markdown 	
	- Generate html pages from Markdown docs: https://github.com/Knagis/CommonMark.NET
	- Markdown UI editor I haven't evaluated yet: https://github.com/nhnent/tui.editor


- ?? TESTING
    - ??Automated UI testing in browser.
    - To catch browser changes that break functionality.
    - Get a quick tool overview.
    - Also can it use diff. browsers and devices?



DIGITAL OCEAN DROPLET REPORT RENDERING TESTING
vbeta 8.0-b0.7 2021-12-28
	WHAT IS THE MAXIMUM wo can print at each gb before swapping occurs and how long does it take		
	*** NOTE NO SWAP FILE FOR THESE TESTS AS DIDN'T KNOW IT WAS NOT SET UP ***

	Basic - Premium AMD Shared CPU 1 vCPU 1 GB 25 GB 1 TB $6/mo
		250wo dispatch report, 33s <---this is it, the max without any swap file, 275 times out forever	
	Basic - Premium AMD Shared CPU 1 vCPU 2 GB 25 GB 2 TB $12/mo
		275, 46s
		300, 50s
		400, 54s
		600, 1m 17s
		1000, 2m 26s (very briefly dipped into swap but only for fractions of a second about 5 times, kept freeing back up again, probably not enough to affect speed)
		1500, abandoned at 7m 16s  (swap city, very slow swapping in an out a lot but no crashing)
		1250, abandoned at 5m (swap city later on but not at first during chrome render, no crash cancelled cleanly)
		1100, it stopped at 2m 53s with timeout dialog but was set to 15m?? WTF, brief swap, expected it to work
		1000, abandoned at 5m (swap city unlike earlier 1000 run, no idea why)
		800, 2m 6s no swap
		900, it stopped at 2m 9s with timeout dialog but set to 15m?? da fuck? 
			weird error at server, put in server todo below to look into 			
	Basic - Premium AMD Shared CPU 2 vCPUs 2 GB 25 GB 3 TB $18/mo $0.027/hr
		(note only diff from above is more cpu, just a test to see if it's actually using them to speed up processing, and check limits)
		800,1m 51s swapped very briefly, cpu showed 130% for chrome process, so maybe chrome using more than one cpu??
		800, just ran again, swapped like made and abandoned at 4m WTF?  Leak?
		trying 200 over and over as a leak test
		200, 34s, 314 pages
		'' same 785 free ram before
		'' 30s, 757 free ram before
		'' 36s, 314 pgs 780 free ram before
	Basic - Premium AMD Shared CPU 2 vCPUs 4 GB 25 GB 4 TB $24/mo $0.036/hr
		200, 39s, 314 pages
		200, 41s, 314 pages
		200, 33s, 314 pages
		1000, 2m 33s 1610 pages (easy peasy, no swap or even low)
		1000, 2m 28s 1610 pages
		1803, cancelled at 6m was swapping hard kswapd0 at 99% nothing happening
		1500, 3m 28s 2404 pages (no swap)		
	Basic - Premium AMD Shared CPU 4 vCPUs 8 GB 25 GB 5 TB $48/mo
		1500, 3m 13s 2404 pages (no swap)
		1803, 3m 37s 2889 pages (holy shit that's a lot)	FIRST run that could handle all without bombing		
		200, 39s, 314 pages	
	----------  WITH 4GB SWAP FILE ENABLED ------
	https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04
	Basic - Premium AMD Shared CPU 1 vCPU 1 GB 25 GB 1 TB $6/mo
	300, 1m 3s, 475 pages (FUCKING WORKS!!!!)
	1000, 3m 33s 1610 pages (FUCK YEAH!)
	1800 timed out at 15 min but no crash