diff --git a/devdocs/deploy.md b/devdocs/deploy.md index f3ab737b..c733346b 100644 --- a/devdocs/deploy.md +++ b/devdocs/deploy.md @@ -1,18 +1,15 @@ -## BUIILD AND DEPLOYMENT - - +# BUIILD AND DEPLOYMENT ### Security scan + If any packages have been changed in the release do a thorough security scan and check of each package - - ### Bump version numbers: Search and replace 8.0.0-beta.5 webapp,server,launcher, v8migrate -Client end ayanova-version.js, +Client end ayanova-version.js, Server end ayanova-version.cs The TWO windows installers are versioned RavenLauncher "single" Launcher is also versioned and needs to be changed individually @@ -22,85 +19,95 @@ Docs index.md file is also versioned #MyAppVersion in C:\data\code\raven\dist\install\windows\x64\standalone.iss ### CLIENT DEV MODE TURNED OFF + Make sure main.js in client devmode is set to false (should just keep it that way unless diagnosing a specific issue during development) ### run build-release.bat + check for errors ## UPLOAD INSTALLERS TO NEXT FOLDER + UPLOAD releases to ayanova.com/download/next folder where it will be staged before officially being released and moved to parent download folder ### Test install to devops server -### Test install to windows server + +### Test install to windows server ### Run e2e tests against Linux and Windows servers + First adjust to the server desired by editing cypress.json and commenting in or out apiBaseUrl and BaseUrl props for local or devops or wherever C:\data\code\raven-test\e2e>npm run all-chrome -(chrome most reliable but *will* fail intermittently so needs multiple runs often) +(chrome most reliable but _will_ fail intermittently so needs multiple runs often) Once tests pass move on to next step ### Run load tests against Linux and Windows servers + (determine if any changes triggered performance issues, needs baseline comparison value) LOAD TESTING STEPS: Login to server to test and seed fresh data first, then can run this test -run C:\data\code\raven-test\load\runtest.bat +run C:\data\code\raven-test\load\runtest.bat Test will take approximately 14 minutes to run Check results in results folder by opening index.html from windows explorer and view in browser BASELINE WHAT TO CHECK: DASHBOARD PAGE is the main page with the main stats: "REquests summary" -The most telling overall stat is the "Requests summary" pie chart pass/fail ratio on the main page -As of today 2021-12-21 release 8-beta.06 the results are PASS: 92.87%, FAIL: 7.13% -Errors *are* expected, it's a stress / load test but there shouldn't be an entirely failing route, just a scattered set of failures, +The most telling overall stat is the "Requests summary" pie chart pass/fail ratio on the main page +As of today 2021-12-21 release 8-beta.06 the results are PASS: 92.87%, FAIL: 7.13% +Errors _are_ expected, it's a stress / load test but there shouldn't be an entirely failing route, just a scattered set of failures, typically unauthorized or bad gateway "Appdex" - skipping this for now, not really relevant at the moment "STATISTICS" table - this is the important one as it shows failures + - Scroll down and look in "executions" -> "FAIL" column and make sure they are not all in one line item if see a lot of failures in a single route need to manually look into that exact issue and route and examine the jmeter test for that route / run it manually perhaps it's normal to see unauthorized or bad gateway errors when the server is super stressed out but other failures might need examination or all of it if less than 90% success - NOTES: + - this also catches reporting as it generates some reports report-render test has explicit report id number for workorder, currently it's 15 for the dispatch report, if all fail it may have changed the id number would be better deterministically but for now this works. - To test local or devops edit the project file "C:\data\code\raven-test\load\stressdevops.jmx" -There are two configuration sections near top for local or devops and it's switched by setting either enabled or disabled: - -or - + There are two configuration sections near top for local or devops and it's switched by setting either enabled or disabled: + + or + - Load test needs fresh seeding so it can create test users. - Once tests pass within expected parameters move on to next step -### MOVE TO PREVIOUS +## MOVE PRIOR VERSION INSTALLERS TO PREVIOUS + Clean out the previous previous download from the previous folder - NOTE: if this is the final major release then it should go in the applicable download folder +NOTE: if this is the final major release then it should go in the applicable download folder Copy the existing downloads to the previous folder preserving (-p) the timestamps Change directory to the previous folder -From the previous folder: cp ../* . -p +From the previous folder: cp ../\* . -p (note: will not copy folders so ok to use) -### DELETE CURRENT DOWNLOADS +### DELETE CURRENT DOWNLOADS (or ignore and overwrite maybe easier and safer anyway) + (OR JUST OVERWRITE THEM IS EASIER) -root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:/var/www/html/ayanova.com/download# rm * - (note: will not remove folders just files by default so ok to use) +root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:/var/www/html/ayanova.com/download# rm \* +(note: will not remove folders just files by default so ok to use) ### COPY INSTALLERS FROM NEXT FOLDER the 4 installer packages found in `ayanova.com/download/next` + - Follow path and folder and url rules in case 4053 https://rockfish.ayanova.com/default.htm#!/rfcaseEdit/4053 - And also v8-migrate.exe if applicable ### Upload static copy of docs to our website + clean out current one from web server: (DONT USE FILEZILLA TO DELETE IT TAKES FUCKING FOREVER) -root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:/var/www/html/ayanova.com/docs# rm * -r - (WATCH THE PATH!!) +root@ubuntu-s-1vcpu-1gb-amd-sfo3-01:/var/www/html/ayanova.com/docs# rm \* -r +(WATCH THE PATH!!) Upload docs: from here: C:\data\code\raven\server\AyaNova\wwwroot\docs\ @@ -110,64 +117,27 @@ to here: /var/www/html/ayanova.com/docs ### profit$$ - - - - - # LINUX MANUAL UPGRADE PROCEDURE From /var/ayanova folder -Download new version (the -O will overwrite so no need to delete existing one): +Download new version (the -O will overwrite so no need to delete existing one): sudo curl -O https://www.ayanova.com/download/ayanova-linux-x64-server.zip Run this one-liner to update and restart: -sudo systemctl stop ayanova.service && rm *.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-linux-x64-server.zip -x "config.json" && sudo systemctl start ayanova.service - - +sudo systemctl stop ayanova.service && rm \*.dll && rm -r ./resource && rm -r ./wwwroot && unzip -o ayanova-linux-x64-server.zip -x "config.json" && sudo systemctl start ayanova.service OLD step by step instructions: sudo systemctl stop ayanova.service -rm *.dll +rm \*.dll rm -r ./resource rm -r ./wwwroot unzip -o ayanova-linux-x64-server.zip -x "config.json" sudo systemctl start ayanova.service +# ========================================================================================================================== - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -========================================================================================================================== -========================================================================================================================== ========================================================================================================================== ========================================= OLD OUTDATED INFO BEYOND HERE ================================================== ========================================================================================================================== @@ -175,6 +145,7 @@ sudo systemctl start ayanova.service ========================================================================================================================== ========================================================================================================================== PUBLIC / INSTALLER + - bump versions, make sure client main.js devmode is false (probably should keep it that way for ongoing use and only enable when dev issue) set version in Server, client and installer iss files - run buildrelease.bat in client project folder @@ -182,78 +153,83 @@ PUBLIC / INSTALLER - Test deploy here first - Post to downloads on ayanova.com - - ### DEPLOY TO DIGITAL OCEAN TEST SERVER PUBLISH TO DEVOPS STEPS FIRST =-=-=- -***** BEFORE RELEASE DOUBLE CHECK CAN RE-SEED FROM CLIENT UI AS THERE IS OFTEN ISSUES WITH DBUTIL ERASE TABLES AND THE ORDERING ETC ****** +**\*** BEFORE RELEASE DOUBLE CHECK CAN RE-SEED FROM CLIENT UI AS THERE IS OFTEN ISSUES WITH DBUTIL ERASE TABLES AND THE ORDERING ETC **\*\*** CLIENT =-=-=-= + - TURN OFF DEV MODE IN MAIN.JS - BUMP PACKAGE.JSON version number -- BUMP CLIENT AyaNova-version.js VERSION NUMBER +- BUMP CLIENT AyaNova-version.js VERSION NUMBER - BUILD RELEASE Run buildrelease.bat in client project folder - SERVER =-=-=- + - BUMP AyaNovaVersion.cs version NUMBER - BUMP AyaNova.csproj version number - BUILD RELEASE Run buildrelease.bat in server project folder - + https://www.ayanova.com/download/ayanova8.alpha.136-win-x64.7z -https://www.ayanova.com/download/migrate136.7z -**ARCHIVE IT, otherwise it could be false positived +https://www.ayanova.com/download/migrate136.7z +\*\*ARCHIVE IT, otherwise it could be false positived - COPY TO DEVOPS SERVER NOTE: if need to replace "files" subfolder on server the rights need to be set to 775 + - Use filezilla to copy files that are new up to server - - Copy to "/home/john/xfer/ayanovadocker/files" - - These two files (and any other changes that are relevant) + - Copy to "/home/john/xfer/ayanovadocker/files" + - These two files (and any other changes that are relevant) - C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\AyaNova.dll - C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\AyaNova.pdb - CONSOLE TO SERVER VIA PUTTY + - Bring down current containers: + - navigate to ~/xfer folder - execute sudo docker-compose down - - Build new image forcing it to update as it sometimes doesn't + - Build new image forcing it to update as it sometimes doesn't + - docker-compose build --force-rm --pull - (NOTE: this will *not* pull newer postgres as it's an image already present so to update that need to do a sudo docker pull postgres:alpine first then build the image - if the major version is incompatible with the old database then there is a whole process to save it by dumping and re-building or just delete all files in the folder /var/lib/ayanova/db rm * -r + (NOTE: this will _not_ pull newer postgres as it's an image already present so to update that need to do a sudo docker pull postgres:alpine first then build the image + if the major version is incompatible with the old database then there is a whole process to save it by dumping and re-building or just delete all files in the folder /var/lib/ayanova/db rm \* -r to completely remove all the db files and the postgres image will make a new empty db on boot) - Run new image - - sudo docker-compose up -d + + - sudo docker-compose up -d - Restart NGINX container (IF NECESSARY) as it seems to sometimes lose it's mind when the AyaNova container is restarted (502 BAD GATEWAY error) + - use the restartnginx.sh script in xfer at the server - or from /docker/letsencrypt-docker-nginx/src/production run sudo docker-compose up -d - Test - If 502 BAD GATEWAY then AyaNova server is not up so the NGINX config bombs because it's proxying to it. - Actually, it just happened and what needs to be done is AyaNova container needs to be running BEFORE nginx container or it seems to get stuck - execute .\restartnginx.sh - - Check logs with sudo docker logs (containerID) to find out what happened + execute .\restartnginx.sh + - Check logs with sudo docker logs (containerID) to find out what happened - Or in some cases (once) Digital Ocean fucked up something - ERASE DB, FETCH LICENSE, GENERATE DATA - ERASE DB: - Stop container if not already stopped: execute sudo docker-compose down - - Edit nano docker-compose.yml, uncomment line with erase db environment variable and re-start to erase db + - Edit nano docker-compose.yml, uncomment line with erase db environment variable and re-start to erase db - sudo docker-compose up -d - Stop the container again, use nano to edit docker-compose.yml and re-comment the erase db environment variable - - Start the container again with the up command + - Start the container again with the up command - FETCH TEST KEY: - Go into the api explorer, authenticate then - - select the POST to license Trial route first { "registeredTo": "TestCo", "emailAddress": "cardjohn@ayanova.com"} + - select the POST to license Trial route first { "registeredTo": "TestCo", "emailAddress": "cardjohn@ayanova.com"} - This seems to setup the db to accept a trial key when fetching the regular key next - select the POST to license route (not the TRIAL one), this will fetch a test key and install it - SEED DB: @@ -261,13 +237,11 @@ https://www.ayanova.com/download/migrate136.7z - NOTE: as of today 2018-10-9 it takes 8 minutes at the Devops server to generate the HUGE dataset - As of today 2019-10-23 it takes 36 minutes which is actually right on track, not sure why it took 8 a year before but likely due to less stuff to do - ### Publish command line: Windows 64 bit: -dotnet publish -o /home/john/Documents/raven/dist/server/win-x64/ -r win-x64 -c Release --self-contained -dotnet publish -o C:\data\code\raven\dist\server\win-x64\ -r win-x64 -c Release --self-contained - +dotnet publish -o /home/john/Documents/raven/dist/server/win-x64/ -r win-x64 -c Release --self-contained +dotnet publish -o C:\data\code\raven\dist\server\win-x64\ -r win-x64 -c Release --self-contained Linux 64 bit: @@ -275,15 +249,15 @@ Normal build without all the .net files (not self contained) This is appropriate for docker based distribution since another image will contain the .net runtime: #### DEFAULT BUILD COMMAND -dotnet publish -o C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\ -c Release + +dotnet publish -o C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\ -c Release (linux) -dotnet publish -o ~/Documents/raven/dist/server/linux-x64/ayanovadocker/files/ -c Release - +dotnet publish -o ~/Documents/raven/dist/server/linux-x64/ayanovadocker/files/ -c Release Self contained (this is appropriate for non containerized distribution, but still requires some Linux native requirements - see below): -dotnet publish -o C:\data\code\raven\dist\server\linux-x64\ -r linux-x64 -c Release --self-contained -dotnet publish -o ~/Documents/raven/dist/server/linux-x64/ -r linux-x64 -c Release --self-contained +dotnet publish -o C:\data\code\raven\dist\server\linux-x64\ -r linux-x64 -c Release --self-contained +dotnet publish -o ~/Documents/raven/dist/server/linux-x64/ -r linux-x64 -c Release --self-contained Needed to change permissions on the AyaNova file to make it executable and also it requires these requirements and probably more: apt-get install libunwind8 @@ -292,13 +266,11 @@ apt-get install libcurl3 //.net core 2.x linux native requirements https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore2x - - Windows 32 bit: -dotnet publish -o /home/john/Documents/raven/dist/server/win-x86/ -r win-x86 -c Release --self-contained +dotnet publish -o /home/john/Documents/raven/dist/server/win-x86/ -r win-x86 -c Release --self-contained Self contained Windows 10 x64: -dotnet publish -o /home/john/Documents/raven/dist/server/win10x64/ -r win10-x64 -c Release --self-contained +dotnet publish -o /home/john/Documents/raven/dist/server/win10x64/ -r win10-x64 -c Release --self-contained PORTABLE RID's: win-x64 @@ -306,43 +278,40 @@ win-x86 linux-x64 //D.O. Linux -ubuntu.16.04-x64 //<--- ends up being the same size as portable linux 64 so not really necessary - +ubuntu.16.04-x64 //<--- ends up being the same size as portable linux 64 so not really necessary - https://docs.microsoft.com/en-us/dotnet/core/deploying/index - https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/index?tabs=aspnetcore2x - https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-publish?tabs=netcore2x - https://docs.microsoft.com/en-us/dotnet/core/rid-catalog +### DOCKER -### DOCKER - - Docker will eat up all the disk space if I don't prune it back once in a while, this seems safe to use and freed up 21GB of space when I ran out recently on devops - PRUNE: https://docs.docker.com/config/pruning/ +Docker will eat up all the disk space if I don't prune it back once in a while, this seems safe to use and freed up 21GB of space when I ran out recently on devops +PRUNE: https://docs.docker.com/config/pruning/ - - Build containers: - - john@debian9John:~/Documents/raven/dist/docker/linux-x64$ docker-compose build - - Run it: - - :~/Documents/raven/dist/docker/linux-x64$ docker-compose up -d - - Build it in prep for running it: - - dotnet publish -o C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\ -c Release - - john@debian9John:~/Documents/raven/server/AyaNova$ dotnet publish -o ~/Documents/raven/dist/docker/linux-x64/ayanovadocker/files -c Release - - - - OPTIONAL SAVING IMAGES (probably will never use this again but keeping for the info) - - Save image: - - docker image save -o .\image\ay-alpha2 gztw/ayanova - - Note: if you use a tag name or repo name it's preserved but if you use an image id it loses the tags - - Not compressed, can be compressed about 60% smaller - - Load image: - - docker image load -i saved_image_file_name_here + - Build containers: + - john@debian9John:~/Documents/raven/dist/docker/linux-x64$ docker-compose build + - Run it: + - :~/Documents/raven/dist/docker/linux-x64$ docker-compose up -d + - Build it in prep for running it: + - dotnet publish -o C:\data\code\raven\dist\docker\linux-x64\ayanovadocker\files\ -c Release + - john@debian9John:~/Documents/raven/server/AyaNova$ dotnet publish -o ~/Documents/raven/dist/docker/linux-x64/ayanovadocker/files -c Release +- OPTIONAL SAVING IMAGES (probably will never use this again but keeping for the info) + - Save image: + - docker image save -o .\image\ay-alpha2 gztw/ayanova + - Note: if you use a tag name or repo name it's preserved but if you use an image id it loses the tags + - Not compressed, can be compressed about 60% smaller + - Load image: + - docker image load -i saved_image_file_name_here #### + - Running docker at our D.O. server - run AyaNova container FIRST sudo docker-compose up -d at ~/xfer/ - To update: - - run a publish command to publish to my local dist/linux-x64/ayanovadocker/files + - run a publish command to publish to my local dist/linux-x64/ayanovadocker/files - Then use Filezilla to copy up to the server at ~/xfer/ayanovadocker/files - Optionally, update the ~/xfer/docker-compose to set a new version number for the image name ("alpha-5" etc or maybe remove the name in future) - If necessary do a docker-compose build to rebuild @@ -351,17 +320,15 @@ ubuntu.16.04-x64 //<--- ends up being the same size as portable linux 64 so not - If necessary can switch to root with command: sudo su - - documented here: https://www.humankode.com/ssl/how-to-set-up-free-ssl-certificates-from-lets-encrypt-using-docker-and-nginx - - ### DOCKER NGINX Let's Encrypt CERTBOT + - https://www.humankode.com/ssl/how-to-set-up-free-ssl-certificates-from-lets-encrypt-using-docker-and-nginx - https://github.com/humankode/letsencrypt-docker-nginx/blob/master/src/production/production.conf - - INITIALLY FETCH CERTIFICATES (MUST START LETSENCRYPT NGINX CONTAINER FIRST AND STOP ALL OTHERS) #### STAGING + sudo docker run -it --rm \ -v /docker-volumes/etc/letsencrypt:/etc/letsencrypt \ -v /docker-volumes/var/lib/letsencrypt:/var/lib/letsencrypt \ @@ -375,6 +342,7 @@ certonly --webroot \ -d helloayanova.com -d www.helloayanova.com -d v8.helloayanova.com -d test.helloayanova.com #### PRODUCTION + sudo docker run -it --rm \ -v /docker-volumes/etc/letsencrypt:/etc/letsencrypt \ -v /docker-volumes/var/lib/letsencrypt:/var/lib/letsencrypt \ @@ -386,9 +354,10 @@ certonly --webroot \ --webroot-path=/data/letsencrypt \ -d helloayanova.com -d www.helloayanova.com -d v8.helloayanova.com -d test.helloayanova.com - #### SAMPLE OUTPUT: + john@ubuntu-s-1vcpu-1gb-sfo2-01:/docker/letsencrypt-docker-nginx/src/letsencrypt$ sudo docker run -it --rm \ + > -v /docker-volumes/etc/letsencrypt:/etc/letsencrypt \ > -v /docker-volumes/var/lib/letsencrypt:/var/lib/letsencrypt \ > -v /docker/letsencrypt-docker-nginx/src/letsencrypt/letsencrypt-site:/data/letsencrypt \ @@ -398,51 +367,49 @@ john@ubuntu-s-1vcpu-1gb-sfo2-01:/docker/letsencrypt-docker-nginx/src/letsencrypt > --email support@ayanova.com --agree-tos --no-eff-email \ > --webroot-path=/data/letsencrypt \ > -d helloayanova.com -d www.helloayanova.com -Saving debug log to /var/log/letsencrypt/letsencrypt.log -Plugins selected: Authenticator webroot, Installer None -Obtaining a new certificate -Performing the following challenges: -http-01 challenge for helloayanova.com -http-01 challenge for www.helloayanova.com -Using the webroot path /data/letsencrypt for all unmatched domains. -Waiting for verification... -Cleaning up challenges +> Saving debug log to /var/log/letsencrypt/letsencrypt.log +> Plugins selected: Authenticator webroot, Installer None +> Obtaining a new certificate +> Performing the following challenges: +> http-01 challenge for helloayanova.com +> http-01 challenge for www.helloayanova.com +> Using the webroot path /data/letsencrypt for all unmatched domains. +> Waiting for verification... +> Cleaning up challenges IMPORTANT NOTES: - - Congratulations! Your certificate and chain have been saved at: - /etc/letsencrypt/live/helloayanova.com/fullchain.pem - Your key file has been saved at: - /etc/letsencrypt/live/helloayanova.com/privkey.pem - Your cert will expire on 2018-06-10. To obtain a new or tweaked - version of this certificate in the future, simply run certbot - again. To non-interactively renew *all* of your certificates, run - "certbot renew" - - Your account credentials have been saved in your Certbot - configuration directory at /etc/letsencrypt. You should make a - secure backup of this folder now. This configuration directory will - also contain certificates and private keys obtained by Certbot so - making regular backups of this folder is ideal. - - If you like Certbot, please consider supporting our work by: - Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate - Donating to EFF: https://eff.org/donate-le +- Congratulations! Your certificate and chain have been saved at: + /etc/letsencrypt/live/helloayanova.com/fullchain.pem + Your key file has been saved at: + /etc/letsencrypt/live/helloayanova.com/privkey.pem + Your cert will expire on 2018-06-10. To obtain a new or tweaked + version of this certificate in the future, simply run certbot + again. To non-interactively renew _all_ of your certificates, run + "certbot renew" +- Your account credentials have been saved in your Certbot + configuration directory at /etc/letsencrypt. You should make a + secure backup of this folder now. This configuration directory will + also contain certificates and private keys obtained by Certbot so + making regular backups of this folder is ideal. +- If you like Certbot, please consider supporting our work by: + Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate + Donating to EFF: https://eff.org/donate-le =-=-=-=-=-=-=-=- DOCKER / LINUX CHEAT SHEET - Alpine linux enter container and run shell from docker: - sudo docker exec -it xfer_ayanova_1 ash +Alpine linux enter container and run shell from docker: +sudo docker exec -it xfer_ayanova_1 ash AYANOVA files are stored in droplet outside docker in /var/lib/ayanova - ######################################################################################## Windows installer worksheet INNOSETUP still the way to go apparently - SIGNING Short story, probably not worth the hassle, needs to come from an authority, expires every year, basically bullshit But if it becomes necessary then: @@ -455,8 +422,8 @@ This guy had to do it and has advice: https://mkaz.blog/code/code-signing-a-windows-application/ This is how to find the sign tool to sign things: -where /R c:\ makecert.* +where /R c:\ makecert.\* https://stackoverflow.com/questions/51418366/makecert-exe-missing-in-windows-10-how-to-get-it-and-use-it Useful looking: -https://blog.jayway.com/2014/09/03/creating-self-signed-certificates-with-makecert-exe-for-development/ \ No newline at end of file +https://blog.jayway.com/2014/09/03/creating-self-signed-certificates-with-makecert-exe-for-development/