# Raven solutions *Solutions, tools and techniques to accomplish goals from research* “Do the simplest thing that will work.” ## HOW TO SCREENSHOTS in manual pages use chrome, zoom in 150% if possible Take screenshots as narrow as possible. If form is too long just take multiple vertically scrolling with whitespace in between, in manual it just looks like a single image ## Middleware docs - https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware/?tabs=aspnetcore2x ## API FEATURES - Ability to set whole api to read only mode by administration or by code like backup routines ### Biz object TAGGER / MARKER INTERFACE or ATTRIBUTES (ITaggable, IAttachable etc etc) - Apparently should use attribute not interfaces: https://stackoverflow.com/questions/2086451/compelling-reasons-to-use-marker-interfaces-instead-of-attributes - https://docs.microsoft.com/en-us/dotnet/standard/attributes/writing-custom-attributes - But if I do use interfaces or need to work with them in future then: - Map all objects on boot: https://garywoodfine.com/get-c-classes-implementing-interface/ - It's called a tagging interface: https://en.wikipedia.org/wiki/Marker_interface_pattern - https://stackoverflow.com/questions/15138924/c-sharp-how-to-determine-if-a-type-implements-a-given-interface ## AUTOMATIC JOB SCHEDULER / RUNNER - jobs in background required for auto backup, mailing, notifications - https://docs.microsoft.com/en-us/dotnet/standard/microservices-architecture/multi-container-microservice-net-applications/background-tasks-with-ihostedservice - Probably don't need a full fledged scheduler because the above should work, but just in case here are some: - Fluent Scheduler looks simpler than hangfire, supports .net core - https://github.com/fluentscheduler/FluentScheduler - Chroniton looks super basic - https://github.com/leosperry/Chroniton/wiki/Example-ASP.NET-Core ## VALIDATION / BUSINESS RULES (FRONT AND BACK) - To run in both places looks like JSON schema is the way to go, it can be stored independent and validated at both ends - It's a mature standard and is platform agnostic - Tutorial from AJV guy: https://code.tutsplus.com/tutorials/validating-data-with-json-schema-part-1--cms-25343 - https://github.com/RSuter/NJsonSchema This generates and validates schema in .net world and is open source MIT license - https://github.com/epoberezkin/ajv This is seemingly the gold standard for javascript based - Lesser things looked at: - https://github.com/cachecontrol/json-rules-engine 128 stars might be adaptable - https://github.com/rsamec/business-rules-engine //this one is for javascript and kind of limited but gives some good ideas ## RESOURCE ACCESS LAYER (DATABASE) - DATABASE - Support Postgresql only out of the box, consider other db's later - CONCURRENCY IS BUILT INTO EFCORE: https://docs.microsoft.com/en-us/ef/core/saving/concurrency - Transactions are also built in by default as the save changes is the only point that stuff actually gets written to the db in most cases - MUST TEST CONCURRENCY AND TRANSACTIONS FAIL - CONTAINERIZED DB's - For development, I don't think there's anything better for databases, it beats manual setup, vagrant boxes, and shared development servers by a long shot. I feel that educating everyone on your team in how to use it is well worth the investment. docker-compose makes setting up even a fairly complicated development environment a breeze. - BACKUP AND RESTORE - DISCOURSE METHOD - I like it because it handles various scenarios and results in a nice SQL command file that rebuilds the whole db, not some cryptic binary format - Can set all api to read only mode, then dumps the db using a db command, then zips it and offers it for download - Backup process - pause the sidekiq background process worker - Can optionally set the api to read only mode (interesting idea) - dumps the data to a file in sql command format for maximum compatibility (even with other db server types puportedly) - Archives it - presents it in the UI for download - unpause the background worker - Restore process - This one is interesting, PGSQL has a "schema" which is a way of partitioning a database to in effect have a separate set of tables in the same db - They move the "public" production "schema" to a "backup" schema (effectively moving it but keeping it in the db) - They restore to a separate interim "restore" "schema" then they move all the tables in the restore schema to the production "public" schema (one by one in a loop) - I guess this way it's reversible if there is an issue but I don't see code to handle any issues - https://github.com/discourse/discourse/tree/master/lib/backup_restore - Also seems to have some capacity to send it to an AWS bitbucket or some thing, maybe an online integration with dropbox or other would be nice MORE DOCKER AND DISCOURSE STUFF https://github.com/discourse/discourse_docker ## ARCHITECTURE / NFR ### Subdomains pointing to differetn droplets tutorial - https://www.digitalocean.com/community/tutorials/how-to-set-up-and-test-dns-subdomains-with-digitalocean-s-dns-panel ### Other stuff This section contains the architectural plans for all aspects of Raven driven by goals and research https://en.wikipedia.org/wiki/Non-functional_requirement - ARchitecture reference resources: - https://stackoverflow.com/questions/5049363/difference-between-repository-and-service-layer - ARCHITECTURE LAYERS - CLIENT - HTML 5 SPA - WEB API LAYER - REPOSITORY LAYER - BUSINESS LAYER (AKA DOMAIN LAYER) - The Business Layer is the place where all the business/domain logic, i.e. rules that are particular to the problem that the application has been built to handle, lives. This might be salary calculations, data analysis modelling, or workflow such as passing a order through different stages. - I'm using this: https://www.thereformedprogrammer.net/a-library-to-run-your-business-logic-when-using-entity-framework-core/ - and this is the repo: https://github.com/JonPSmith/EfCore.GenericBizRunner - DATA ACCESS LAYER (EF CORE) - EF CORE multiple database stuff - Migrations with different providers: https://stackoverflow.com/questions/42819371/ef-core-multiple-migration-sets - DATABASE - .NET CORE DEPLOYMENT - Basically just get the files on to the system in one case or need .net installed as a pre-requisite then drop the files on - https://docs.microsoft.com/en-us/dotnet/core/deploying/deploy-with-cli - https://docs.microsoft.com/en-us/dotnet/core/deploying/ - KESTREL alone - Kestrel alone can be used but it won't work if need to share a port with something else and differentiate by host header - in case of host header issue can run NGinx in front or IIS - More to come here once we have a testable skeleton project to set up - STATIC FILE CACHING - https://andrewlock.net/adding-cache-control-headers-to-static-files-in-asp-net-core/ - Asp.net Core 2.0 application stack - .net core WEBAPI project - Swagger for documentation - REST best practices - Excellent reference guide here: https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md - URLS: A good api url: https://api.ayanova.com/v1.0/customer/22 - Keep length under 2000 characters for maximum client compatibility - concurrency (The ETag response-header field provides the current value of the entity tag for the requested variant. Used with If-Match, If-None-Match and If-Range to implement optimistic concurrency control.) - JSON property names SHOULD be camelCased. - There are specific Date, time, Duration, Interval formats that should be used (https://github.com/Microsoft/api-guidelines/blob/vNext/Guidelines.md#113-json-serialization-of-dates-and-times) - Error response: The error response MUST be a single JSON object. This object MUST have a name/value pair named "error." The value MUST be a JSON object. This object MUST contain name/value pairs with the names "code" and "message," and it MAY contain name/value pairs with the names "target," "details" and "innererror." eg: error:{code=1200,message="blah"} error:{code=1200,message="blah",target="eg property name", details="details for programmer", innererror:} - Versioning: this is an area I need to treat carefully, there are tools to make it easier: - I will use URL Path segment versioning, i.e. api/v1.0/customer/22 - https://www.hanselman.com/blog/ASPNETCoreRESTfulWebAPIVersioningMadeEasy.aspx - https://github.com/Microsoft/aspnet-api-versioning/wiki - https://github.com/Microsoft/aspnet-api-versioning/tree/master/samples/aspnetcore/SwaggerSample - Push notifications (I.e. on a customer record being created or a workorder updated or...?) - If so there is a segment in the rest doc from microsoft that goes over it in detail - API THROTTLING / RATE LIMITING - https://github.com/stefanprodan/AspNetCoreRateLimit - ASYNC / BUSINESS LAYER - https://stackoverflow.com/questions/42276149/best-practice-for-using-async-await-in-webapi - https://stackoverflow.com/questions/41719661/asp-net-core-making-service-asynchronous - http://www.codemag.com/article/1701061 BUSINESS LAYER STUFF - https://github.com/RickStrahl/AlbumViewerVNext ## AUTOMAPPER - Sounds like a tool I might need, here is a good tutorial: https://dotnetcoretutorials.com/2017/09/23/using-automapper-asp-net-core/ ## TESTING - TESTING performance / load / resiliency testing - https://docs.microsoft.com/en-us/aspnet/core/testing/ - Spend the time to generate realistic production level data in large quantity for testing - Do not do any integration testing without realistic data - Data generation for testing could be re-used for trial data generation for customer evaluation purposes - Test with production sized data !!!! (did not do this properly when doing AyaNova originally) - TOOLS - xUnit https://xunit.github.io/ - Mocking, different test runners here: http://asp.net-hacker.rocks/2017/03/31/unit-testing-with-dotnetcore.html - GENFU - tries to automatically fill objects which sounds dicey https://github.com/MisterJames/GenFu/ - https://github.com/bchavez/Bogus This is a .net port of faker.js and as such probably a good choice - Testing with a docker container (old but interesting) https://devblog.xero.com/getting-started-with-running-unit-tests-in-net-core-with-xunit-and-docker-e92915e4075c ## LOGGING - Watch out, logging can be a huge performance drain, test with logs on and off to see what and if necessary replace logging class with something faster. ## REFERENCE RESOURCES - ASP.NET CORE FUNDAMENTALS DOCS - Everything to do with asp.net core fundamental coding aspects - https://docs.microsoft.com/en-us/aspnet/core/fundamentals/?tabs=aspnetcore2x - AWESOME .NET CORE - Has a list of solutions for .net core project needs - https://github.com/thangchung/awesome-dotnet-core - THIS TYPE OF PROJECT REFERENCE CODE - NICE DIAGRAM OF OUR ARCHITECTURE FOR RAVEN: http://www.dotnetcurry.com/entityframework/1348/ef-core-web-api-crud-operations - https://github.com/RickStrahl/AlbumViewerVNext - This is a good one, it uses .net core, runs on multiple platforms, has an angular front end, is from a seasoned practical developer etc etc - https://github.com/dodyg/practical-aspnetcore - .net samples and tutorials on official docs page - https://docs.microsoft.com/en-us/dotnet/samples-and-tutorials/ - https://samueleresca.net/2017/02/implementing-solid-data-access-layers-using-asp-net-core/ - AUTOMAPPER: https://github.com/AutoMapper/AutoMapper/wiki/Getting-started - https://www.infragistics.com/community/blogs/dhananjay_kumar/archive/2016/03/07/how-to-implement-the-repository-pattern-in-asp-net-mvc-application.aspx - ORCHARD - Almost the perfect reference application, they are doing what I will be doing but for a CMS - https://github.com/OrchardCMS/OrchardCore - This samples link actually contains a lot of useful info as well like multi-tenanting stuff etc - https://github.com/OrchardCMS/OrchardCore.Samples - DISCOURSE - this app is kind of what raven will become in many ways architecturally, - It's a message board that is modern uses Postgresql, digital ocean docker container, mobile and desktop browser UI - It's open source so plunder it's source code here: - https://github.com/discourse/discourse - https://github.com/discourse/discourse_docker/blob/master/samples/standalone.yml - DOCKER CONTAINERIZED APP BUILD GUIDE - This guide has a *lot* of good info in it that is right up my alley: - https://www.red-gate.com/simple-talk/sysadmin/containerization/overcoming-challenges-microservices-docker-containerisation/?utm_source=simpletalk&utm_medium=pubemail&utm_content=20170926-slota5&utm_term=simpletalkmain - FRONT END DASHBOARD - Cool dashboard with graphs and shit: https://github.com/tabler/tabler?utm_source=DigitalOcean_Newsletter ## HELP AND DOCUMENTATION RESOURCES - ONLINE HELP FOR RAVEN CUSTOMERS - The Jenkins help page has a really good layout for help with Guided Tour, User Handbook, Resources and Recent Tutorials on left panel - https://jenkins.io/doc/pipeline/tour/hello-world/ ## GRAPHICS / ARTWORK UI FANCIFICATION RESOURCES - Free for use background image generator that looks really nice and soothing: https://coolbackgrounds.io/ - Free for any use even without attribution stock photography: https://unsplash.com/ ## ASCII ART //http://www.patorjk.com/software/taag/#p=display&f=ANSI%20Shadow&t=PM ## Time zone stuff //play with any time zone with codes http://www.timezoneconverter.com/cgi-bin/zoneinfo //list of time zone offsets https://en.wikipedia.org/wiki/List_of_UTC_time_offsets Good test time zones: America/St_Johns - because it has a fractional hour difference America/New_York America/Los_Angeles ## How to make a postgres portable from binaries download latest binaries: https://www.enterprisedb.com/download-postgresql-binaries unzip latest binaries to a source folder make a dest folder in c:\data\code\postgres_xx copy bin, lib, share folders from source to dest Create a new db, run this once: c:\data\code\postgres_14\bin\initdb -D ^"C^:^\data^\code^\postgres^_14^\ayanova^" -U postgres -A trust that's it, run it with: C:\data\code\postgres_14\bin\pg_ctl -D ^"C^:^\data^\code^\postgres^_14^\ayanova^" -l logfile start Note: for test win64 build just copy the empty db files generated above (before any access is done) into the C:\data\code\raven\dist\win-x64\data\database folder ## How to find the postgres data directory sudo -u postgres psql -c "show data_directory;" ## PSQL to nose around ayanova database sudo -u postgres psql https://www.postgresqltutorial.com/postgresql-administration/psql-commands/ ## How fast is ayanova across the internet TEST: TEST digitalocean, set up a server in every data center far away from here, test ayanvoa on each one and see how peformant it is perceptually, i.e. does it even matter which server we're locating people on?? Bangalore - seems almost undetectably the same as sanfran todo: test entering data on a work order, open an existing, create etc Singapor - excellent speed no appreciable different Frankfurt - excellent speed as well RESULT: I'd say for AyaNova there is no appreciable different from one data center to the next, the delay just isn't enough to really factor for biz software and it's super efficent in traffic which helps (back pat to me for thinking efficiently in advance) ## How to point a subdomain from an external company's domain to one of our hosted ayanova instances and have it work with letsencrypt etc e.g. ayanova.thecompany.com pointing to thecompany.ayanova.com RESEARCH: how to enable a portion of a domain from an outside to point to their droplet, is that on them or on us? how does discourse do it?? https://meta.discourse.org/t/configure-your-domain-name-for-hosted-discourse/21827 SO, I would say it works just like we do with the test servers, they would need to edit their domain record to point a subdomain to us so we would tell them to do that, i.e. they would say aya.fouralarm.ca points to [ipaddressofdroplet] or discourse does it by subdomain, they would say point to fouralarm.hosted-by-discourse.com so I wonder if we can do that too, Maybe a way to test this out is like this: make a droplet, make a domain record gztestco.helloayanova.com that points to that droplet then make a cert on the droplet for the original gztestco.helloayanova.com and in nginx, request the cert so this is like the initial trial period or setup. Test, confirm gztestco.helloayanova.com is working then go to ayanova domain and make a CNAME (CNAME is different and is host pointing to host not to ip address) record aya.ayanova.com and point it to gztestco.helloayanova.com Then change nginx config in droplet and ADD aya.ayanova.com and then request a new cert with both aya.ayanova.com and gztestco.helloayanova.com and see if it works from both domains. ** To request a new cert need different command and need to know cert name which is the first domain requested but to confirm use certbot certificates to view the name then use the command like this pattern: certbot --nginx --cert-name gztestco.helloayanova.com -d gztestco.helloayanova.com,aya.ayanova.com (we would keep both domains in case they have dns issues with their own domain and need access or fuck up the redirect somehow later on) THIS works, and this is the pattern for customers where aya.ayanova.com would be their equivalent to their ayanova subdomain record they make with their domain registrar or whoever. **TESTED, WORKS!: I do have a concern about the nginx cert autorenewal as expanding the domains didn't automatically edit the nginx config file and add the managed by certbot bit for the aya.ayanova.com domain ## How to make test sales and product links in ShareIt fake sale of v8 to get license emails to update rockfish to auto read in the deets FAKE CREDIT CARD I have generated a test credit card number for your account. Please use the following as the credit card number making sure to use the word test and the 12 digit number with no spaces. You can use any valid information you would like for the other fields. test134661442658 Expiration Date: any date in the future CVS code: 123 Example typical database ID: tZy9aSHjacFf9a9v3EtzUqM1amq/eMvJa2nXswLle74= Links to purchase subscription month to month https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B301028467%5D=1 subscription yearly https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B301028468%5D=1 Perpetual single license: https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B301028314%5D=1 *** TESTING THIS ONE FIRST Perpetual single one year maintenance ACTIVE DISCOUNT PRICE: https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B301028315%5D=1 Perpetual single one year maintenance NEW FULL PRICE: https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B301028317%5D=1 v7 single for comparison 300740315 https://order.mycommerce.com/cart/new?vendorid=14466&PRODUCT%5B300740315%5D=1 Address to use for testing 05-3610 Christie Parkway Courtenay B.C. Canada V9J 9T6 { "creation_date": "2022-08-29T00:37:59Z", "id": 349316690, "order_notification": { "purchase": { "customer_data": { "billing_contact": { "address": { "city": "Courtenay", "country": "Canada", "country_id": "CA", "postal_code": "V9J 9T6", "state": "British Columbia", "state_id": "BC", "street1": "05-3610 Christie Parkway" }, "company": "GZ TestCo", "email": "gzmailadmin@gmail.com", "first_name": "Test", "last_name": "Testerson" }, "customer_payment_data": { "currency": "CAD", "payment_method": "Other" }, "delivery_contact": { "address": { "city": "Courtenay", "country": "Canada", "country_id": "CA", "postal_code": "V9J 9T6", "state": "British Columbia", "state_id": "BC", "street1": "05-3610 Christie Parkway" }, "company": "GZ TestCo", "email": "gzmailadmin@gmail.com", "first_name": "Test", "last_name": "Testerson" }, "language": "English", "language_iso": "en", "reg_name": "GZ TestCo", "shopper_id": "64475647", "subscribe_newsletter": false, "user_id": "gzmailadmin@gmail.com-32" }, "is_test": true, "payment_complete_date": "2022-08-29T00:37:59Z", "payment_status": "testpaymentarrived", "payment_status_id": "TCA", "purchase_date": "2022-08-29T00:37:59Z", "purchase_id": 813793143, "purchase_item": [ { "additional_information": [ { "additional_id": "AGGREENOREFUNDS", "additional_value": "YES" }, { "additional_id": "AGREEPAYMETHODVALIDCANCEL", "additional_value": "YES" }, { "additional_id": "AGREEEXPIRESIFNOTPAID", "additional_value": "YES" }, { "additional_id": "DATABASEID", "additional_value": "tZy9aSHjacFf9a9v3EtzUqM1amq/eMvJa2nXswLle74=" } ], "currency": "USD", "delivery_type": "Electronically", "discount": 0.0, "extended_download_price": 0.0, "manual_order_price": 0.0, "notification_no": 0, "product_id": 301028315, "product_name": "Single AyaNova service technician 1 year maintenance plan - active", "product_single_price": 100.0, "purchase_item_key": [], "quantity": 5, "running_no": 1, "shipping_price": 0.0, "shipping_vat_pct": 0.0, "subscription": { "expiration_date": "2023-08-29T00:37:59Z", "id": "813793143-1", "interval": "Yearly without end", "renewal_discount_count": "", "renewal_discount_start": "", "renewal_type": "auto", "retention_discount_count": "", "retention_discount_percent": "", "start_date": "2022-08-29T00:00:00", "status": "ToProcess", "status_id": "TOP" }, "vat_pct": 12.0, "your_product_id": "maintenanceplanactive" } ], "purchase_origin": "online" } } }