raven/devdocs/solutions.txt

# Raven solutions 
*Solutions, tools and techniques to accomplish goals from research*

“Do the simplest thing that will work.”



## Middleware docs
    - https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/?tabs=aspnetcore2x

## API FEATURES
    - Ability to set whole api to read only mode by administration or by code like backup routines

### Biz object TAGGER / MARKER INTERFACE or ATTRIBUTES (ITaggable, IAttachable etc etc)
    - Apparently should use attribute not interfaces: https://stackoverflow.com/questions/2086451/compelling-reasons-to-use-marker-interfaces-instead-of-attributes
        - https://docs.microsoft.com/en-us/dotnet/standard/attributes/writing-custom-attributes
    - But if I do use interfaces or need to work with them in future then:
        - Map all objects on boot: https://garywoodfine.com/get-c-classes-implementing-interface/
        - It's called a tagging interface: https://en.wikipedia.org/wiki/Marker_interface_pattern
        - https://stackoverflow.com/questions/15138924/c-sharp-how-to-determine-if-a-type-implements-a-given-interface



## AUTOMATIC JOB SCHEDULER / RUNNER
    - jobs in background required for auto backup, mailing, notifications
    - https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/multi-container-microservice-net-applications/background-tasks-with-ihostedservice
  
    - Probably don't need a full fledged scheduler because the above should work, but just in case here are some:
        - Fluent Scheduler looks simpler than hangfire, supports .net core
            - https://github.com/fluentscheduler/FluentScheduler
        - Chroniton looks super basic
            - https://github.com/leosperry/Chroniton/wiki/Example-ASP.NET-Core


## VALIDATION / BUSINESS RULES (FRONT AND BACK)
 - To run in both places looks like JSON schema is the way to go, it can be stored independent and validated at both ends
 - It's a mature standard and is platform agnostic
 - Tutorial from AJV guy: https://code.tutsplus.com/tutorials/validating-data-with-json-schema-part-1--cms-25343
 - https://github.com/RSuter/NJsonSchema This generates and validates  schema in .net world and is open source MIT license
 - https://github.com/epoberezkin/ajv This is seemingly the gold standard for javascript based
 - Lesser things looked at:
 - https://github.com/cachecontrol/json-rules-engine 128 stars might be adaptable
 - https://github.com/rsamec/business-rules-engine //this one is for javascript and kind of limited but gives some good ideas


## RESOURCE ACCESS LAYER (DATABASE)
    - DATABASE       
        - Support Postgresql only out of the box, consider other db's later
    - CONCURRENCY IS BUILT INTO EFCORE: https://docs.microsoft.com/en-us/ef/core/saving/concurrency
        - Transactions are also built in by default as the save changes is the only point that stuff actually gets written to the db in most cases
        - MUST TEST CONCURRENCY AND TRANSACTIONS FAIL
    - CONTAINERIZED DB's
        - For development, I don't think there's anything better for databases, it beats manual setup, vagrant boxes, and shared development servers by a long shot. I feel that educating everyone on your team in how to use it is well worth the investment. docker-compose makes setting up even a fairly complicated development environment a breeze.
    - BACKUP AND RESTORE
        - DISCOURSE METHOD  
            - I like it because it handles various scenarios and results in a nice SQL command file that rebuilds the whole db, not some cryptic binary format      
        - Can set all api to read only mode, then dumps the db using a db command, then zips it and offers it for download
        - Backup process 
            - pause the sidekiq background process worker
            - Can optionally set the api to read only mode (interesting idea)
            - dumps the data to a file in sql command format for maximum compatibility (even with other db server types puportedly)
            - Archives it
            - presents it in the UI for download
            - unpause the background worker
        - Restore process
            - This one is interesting, PGSQL has a "schema" which is a way of partitioning a database to in effect have a separate set of tables in the same db
            - They move the "public" production "schema" to a "backup" schema (effectively moving it but keeping it in the db)
            - They restore to a separate interim "restore" "schema" then they move all the tables in the restore schema to the production "public" schema (one by one in a loop)
            - I guess this way it's reversible if there is an issue but I don't see code to handle any issues
        - https://github.com/discourse/discourse/tree/master/lib/backup_restore
        - Also seems to have some capacity to send it to an AWS bitbucket or some thing, maybe an online integration with dropbox or other would be nice
    MORE DOCKER AND DISCOURSE STUFF
        https://github.com/discourse/discourse_docker



## ARCHITECTURE / NFR





### Subdomains pointing to differetn droplets tutorial
    - https://www.digitalocean.com/community/tutorials/how-to-set-up-and-test-dns-subdomains-with-digitalocean-s-dns-panel

### Other stuff
This section contains the architectural plans for all aspects of Raven driven by goals and research
https://en.wikipedia.org/wiki/Non-functional_requirement

    - ARchitecture reference resources:
        - https://stackoverflow.com/questions/5049363/difference-between-repository-and-service-layer

    - ARCHITECTURE LAYERS 
        - CLIENT 
            - HTML 5 SPA  
        - WEB API LAYER
        - REPOSITORY LAYER
        - BUSINESS LAYER (AKA DOMAIN LAYER)
            - The Business Layer is the place where all the business/domain logic, i.e. rules that are particular to the problem 
            that the application has been built to handle, lives. This might be salary calculations, data analysis modelling, 
            or workflow such as passing a order through different stages.
            - I'm using this: https://www.thereformedprogrammer.net/a-library-to-run-your-business-logic-when-using-entity-framework-core/
            - and this is the repo: https://github.com/JonPSmith/EfCore.GenericBizRunner
            
        - DATA ACCESS LAYER (EF CORE) 
            - EF CORE multiple database stuff
                - Migrations with different providers: https://stackoverflow.com/questions/42819371/ef-core-multiple-migration-sets
        - DATABASE 




    - .NET CORE DEPLOYMENT
        - Basically just get the files on to the system in one case or need .net installed as a pre-requisite then drop the files on
        - https://docs.microsoft.com/en-us/dotnet/core/deploying/deploy-with-cli
        - https://docs.microsoft.com/en-us/dotnet/core/deploying/
    - KESTREL alone 
        - Kestrel alone can be used but it won't work if need to share a port with something else and differentiate by host header
            - in case of host header issue can run NGinx in front or IIS
        - More to come here once we have a testable skeleton project to set up    
    - STATIC FILE CACHING
        - https://andrewlock.net/adding-cache-control-headers-to-static-files-in-asp-net-core/

    - Asp.net Core 2.0 application stack
    - .net core WEBAPI project
    - Swagger for documentation

- REST best practices
    - Excellent reference guide here: https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md
    - URLS: A good api url: https://api.ayanova.com/v1.0/customer/22  
        - Keep length under 2000 characters for maximum client compatibility
        - concurrency (The ETag response-header field provides the current value of the entity tag for the requested variant. Used with If-Match, If-None-Match and If-Range to implement optimistic concurrency control.)
    - JSON property names SHOULD be camelCased.
    - There are specific Date, time, Duration, Interval formats that should be used (https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#113-json-serialization-of-dates-and-times)
    - Error response: The error response MUST be a single JSON object. This object MUST have a name/value pair named "error." The value MUST be a JSON object.
    This object MUST contain name/value pairs with the names "code" and "message," and it MAY contain name/value pairs with the names "target," "details" and "innererror."
    eg: error:{code=1200,message="blah"}  error:{code=1200,message="blah",target="eg property name", details="details for programmer", innererror:}
    - Versioning: this is an area I need to treat carefully, there are tools to make it easier: 
        - I will use URL Path segment versioning, i.e. api/v1.0/customer/22
        - https://www.hanselman.com/blog/ASPNETCoreRESTfulWebAPIVersioningMadeEasy.aspx
        - https://github.com/Microsoft/aspnet-api-versioning/wiki 
        - https://github.com/Microsoft/aspnet-api-versioning/tree/master/samples/aspnetcore/SwaggerSample    
    - Push notifications  (I.e. on a customer record being created or a workorder updated or...?) 
        - If so there is a segment in the rest doc from microsoft that goes over it in detail
    - API THROTTLING / RATE LIMITING
        - https://github.com/stefanprodan/AspNetCoreRateLimit
    - ASYNC / BUSINESS LAYER
        - https://stackoverflow.com/questions/42276149/best-practice-for-using-async-await-in-webapi
        - https://stackoverflow.com/questions/41719661/asp-net-core-making-service-asynchronous
        - http://www.codemag.com/article/1701061 BUSINESS LAYER STUFF
        - https://github.com/RickStrahl/AlbumViewerVNext 

## AUTOMAPPER
    - Sounds like a tool I might need, here is a good tutorial: https://dotnetcoretutorials.com/2017/09/23/using-automapper-asp-net-core/



## TESTING 
- TESTING performance / load / resiliency testing
    - https://docs.microsoft.com/en-us/aspnet/core/testing/
    - Spend the time to generate realistic production level data in large quantity for testing
    - Do not do any integration testing without realistic data
    - Data generation for testing could be re-used for trial data generation for customer evaluation purposes
    - Test with production sized data !!!! (did not do this properly when doing AyaNova originally)
    - TOOLS
        - xUnit https://xunit.github.io/
        - Mocking, different test runners here: http://asp.net-hacker.rocks/2017/03/31/unit-testing-with-dotnetcore.html
        - GENFU - tries to automatically fill objects which sounds dicey https://github.com/MisterJames/GenFu/
        - https://github.com/bchavez/Bogus  This is a .net port of faker.js and as such probably a good choice
        - Testing with a docker container (old but interesting) https://devblog.xero.com/getting-started-with-running-unit-tests-in-net-core-with-xunit-and-docker-e92915e4075c
## LOGGING
    - Watch out, logging can be a huge performance drain, test with logs on and off to see what and if necessary replace logging class with something faster.




## REFERENCE RESOURCES


- ASP.NET CORE FUNDAMENTALS DOCS
    - Everything to do with asp.net core fundamental coding aspects
    - https://docs.microsoft.com/en-us/aspnet/core/fundamentals/?tabs=aspnetcore2x

- AWESOME .NET CORE
    - Has a list of solutions for .net core project needs
    - https://github.com/thangchung/awesome-dotnet-core


- THIS TYPE OF PROJECT REFERENCE CODE 
    - NICE DIAGRAM OF OUR ARCHITECTURE FOR RAVEN: http://www.dotnetcurry.com/entityframework/1348/ef-core-web-api-crud-operations
    - https://github.com/RickStrahl/AlbumViewerVNext
        - This is a good one, it uses .net core, runs on multiple platforms, has an angular front end, is from a seasoned practical developer etc etc
    - https://github.com/dodyg/practical-aspnetcore
    - .net samples and tutorials on official docs page 
        - https://docs.microsoft.com/en-us/dotnet/samples-and-tutorials/
    - https://samueleresca.net/2017/02/implementing-solid-data-access-layers-using-asp-net-core/
        - AUTOMAPPER: https://github.com/AutoMapper/AutoMapper/wiki/Getting-started
    - https://www.infragistics.com/community/blogs/dhananjay_kumar/archive/2016/03/07/how-to-implement-the-repository-pattern-in-asp-net-mvc-application.aspx
        

- ORCHARD
	- Almost the perfect reference application, they are doing what I will be doing but for a CMS
	- https://github.com/OrchardCMS/OrchardCore
	- This samples link actually contains a lot of useful info as well like multi-tenanting stuff etc
	- https://github.com/OrchardCMS/OrchardCore.Samples 
	
- DISCOURSE
    - this app is kind of what raven will become in many ways architecturally,
    - It's a message board that is modern uses Postgresql, digital ocean docker container, mobile and desktop browser UI 
    - It's open source so plunder it's source code here:
    - https://github.com/discourse/discourse
    - https://github.com/discourse/discourse_docker/blob/master/samples/standalone.yml
	
	

- DOCKER CONTAINERIZED APP BUILD GUIDE
    - This guide has a *lot* of good info in it that is right up my alley:
    - https://www.red-gate.com/simple-talk/sysadmin/containerization/overcoming-challenges-microservices-docker-containerisation/?utm_source=simpletalk&utm_medium=pubemail&utm_content=20170926-slota5&utm_term=simpletalkmain

- FRONT END DASHBOARD

    - Cool dashboard with graphs and shit: https://github.com/tabler/tabler?utm_source=DigitalOcean_Newsletter

## HELP AND DOCUMENTATION RESOURCES
    - ONLINE HELP FOR RAVEN CUSTOMERS
        - The Jenkins help page has a really good layout for help with Guided Tour, User Handbook, Resources and Recent Tutorials on left panel
        - https://jenkins.io/doc/pipeline/tour/hello-world/

   
## GRAPHICS / ARTWORK UI FANCIFICATION RESOURCES
- Free for use background image generator that looks really nice and soothing: https://coolbackgrounds.io/
- Free for any use even without attribution stock photography: https://unsplash.com/ 


## ASCII ART
//http://www.patorjk.com/software/taag/#p=display&f=ANSI%20Shadow&t=PM

## Time zone stuff
//play with any time zone with codes
http://www.timezoneconverter.com/cgi-bin/zoneinfo

//list of time zone offsets
https://en.wikipedia.org/wiki/List_of_UTC_time_offsets

Good test time zones:
America/St_Johns - because it has a fractional hour difference
America/New_York 
America/Los_Angeles


## How to make a postgres portable from binaries
	download latest binaries: https://www.enterprisedb.com/download-postgresql-binaries
	unzip latest binaries to a source folder
	make a dest folder in c:\data\code\postgres_xx
	copy bin, lib, share folders from source to dest
	Create a new db, run this once: 
	c:\data\code\postgres_14\bin\initdb -D ^"C^:^\data^\code^\postgres^_14^\ayanova^" -U postgres -A trust
	that's it, run it with: 
	C:\data\code\postgres_14\bin\pg_ctl -D ^"C^:^\data^\code^\postgres^_14^\ayanova^" -l logfile start

    Note: for test win64 build just copy the empty db files generated above (before any access is done) into the C:\data\code\raven\dist\win-x64\data\database folder