admin/rockfish
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Browse Source
This commit is contained in:
John Cardinal
2022-09-24 00:22:42 +00:00
parent a4c6492c8b
commit 1159faf0f9
1 changed files with 304 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files

507
docs/docs/howto.md
View File

@@ -1,20 +1,310 @@
# EMAIL FROM SERVER
# How to
Common server management and troubleshooting tasks
## LETSENCRYPT
### Expanding / changing certificate
Changing a Certificate’s Domains
The --cert-name flag can also be used to modify the domains a certificate contains, by specifying new domains using the -d or --domains flag.
If certificate example.com previously contained example.com and www.example.com, it can be modified to only contain example.com
by specifying only example.com with the -d or --domains flag. Example:
```bash
certbot certonly --cert-name example.com -d example.com
```
The same format can be used to expand the set of domains a certificate contains, or to replace that set entirely:
```bash
certbot certonly --cert-name example.com -d example.org,www.example.org
```
SO FOR EXAMPLE for HelloAyaNova.com
```bash
certbot --nginx --cert-name helloayanova.com -d test.helloayanova.com,helloayanova.com,www.helloayanova.com,1665.helloayanova.com,7331.helloayanova.com,8888.helloayanova.com,a261.helloayanova.com
```
## NGINX
### Log
Just view logs:
```bash
less /var/log/nginx/error.log
less /var/log/nginx/access.log
```
Dynamically show errors and access logs using tail command:
```bash
tail -f /var/log/nginx/error.log
tail -f /var/log/nginx/access.log
```
USEFUL COMMANDS
Get NGINX version from containerized nginx:
docker container exec -it \[container-id\] nginx -v
SUDO -> ROOT for session
sudo -i
(use exit to go back to john account)
Connect shell to running container:
docker exec -it \[container-id\] bash (or ash on alpine)
CERTBOT DRY RUN
append --dry-run to command
BACKUP FOLDERS AND FILES
cp -R pecklist pecklist_backup
REMOVE OLD .NET RUNTIMES:
Some of these commands get the job done
cd /usr/share/dotnet/shared
apt-cache pkgnames Microsoft\*
apt-cache pkgnames
apt list --installed
apt-get remove dotnet-runtime-3.0
## APT package manager
### Kept back package
If a package has been "kept back":
https://superuser.com/questions/1107334/apt-says-packages-have-been-kept-back-what-to-do/1108268
```bash
apt list --upgradable
sudo apt --with-new-pkgs upgrade
```
### System restart required check
```bash
cat /var/run/reboot-required
```
### Uninstall and purge package
sudo apt-get --purge remove package_name
apt-get autoremove
## SPACES BACKUP
Setup spaces utilities:
s3cmd: https://www.digitalocean.com/docs/spaces/resources/s3cmd/
manually copy a file to spaces:
```bash
/usr/bin/s3cmd put filename.zip s3://gztw1
```
## POSTGRES
### SERVICES
```bash
sudo systemctl stop postgresql
sudo systemctl start postgresql
sudo systemctl restart postgresql
sudo systemctl status postgresql
```
View edit service files
```bash
nano /etc/systemd/system/ayanova.service
```
### PSQL RUN SQL COMMANDS
Commands must be terminated with a semicolon or it will wait until it sees one
Open sql console as postgres admin user:
sudo -u postgres psql
#### Restore AyaNova from backup
- Ensure have the db-xxxxxx.backup database backup and the at-xxxxx.zip attachments file
- Make a backup of the data folders first for insurance
- restart ayanova with permanently erase database set, this will erase the db adn also all attachments
- Stop the AyaNova server `systemctl stop ayanova.service`
- Upload the attachments to the root home folder
- Upload the database backup to `/var/lib/postgresql`
- Unzip attachments: from ayanova data attachments folder run `unzip ~/at-20220920220001750`
- Restore the database from anywhere with this command: `sudo -u postgres pg_restore -U postgres -c -d AyaNova /var/lib/postgresql/db-20220920220000302.backup`
- Start the AyaNova server `systemctl start ayanova.service`
- confirm can login all is well
#### List all dbs
`\l`
#### Get disk size of db
In bytes:
`select pg_database_size('AyaNova');`
Prettified to human readable sizes:
```sql
SELECT
pg_size_pretty (
pg_database_size ('AyaNova')
);
```
Display the size of _all_ databases in human readable format
```sql
SELECT
pg_database.datname,
pg_size_pretty(pg_database_size(pg_database.datname)) AS size
FROM pg_database;
```
## LOG FILES
show log file usage of system logs
```bash
root@devops:/var/ayanova/data/logs# du -h /var/log/
```
### Cleaning out log files
https://support.hostway.com/hc/en-us/articles/360001972270-How-to-clean-log-files-in-Linux
https://linuxhandbook.com/clear-systemd-journal-logs/
### Linux system journal logs
```bash
journctl
```
use g key to go to start and G to go to end and q to quit
#### capping size
If down the road need to cap system journal logs size:
Add this to the standard server config script
cap system journal logs in linux to 250mb or some reasonable value
https://linuxhandbook.com/clear-systemd-journal-logs/
### Viewing huge log files
Use Less command, it has special commands to jump to start and end of file g G = go
q to quit
page up down and more if look online
## FIND BIG FILES
Find the biggest folders from any directory:
```bash
root@devops:/# du -h --max-depth=1 | sort -rh
```
Then go into each folder and run again to limit to that folder and subfolders
## DISK USAGE
```bash
df -h
```
## SSH KEYS
### Add existing user manually
(joeblogs is the user folder here, to create user need to use adduser add to sudo group etc, lots of guides online for that)
### Create the .ssh directory, and set its permissions
```bash
mkdir ~joebloggs/.ssh
chown joebloggs ~joebloggs/.ssh
chmod 0700 ~joebloggs/.ssh
```
### Create the authorized keys file, and set its permissions
```bash
nano ~joebloggs/.ssh/authorized_keys <--- Here, paste in the key from digitalocean keys in account settings security
chown joebloggs ~joebloggs/.ssh/authorized_keys
chmod 0700 ~joebloggs/.ssh/authorized_keys
```
### How we use keys
All servers use unique ssh key only stored in keepass (stairs) and accessed by using KeeAgent in windows and firewalled to only the home ip address
In a pinch the keys could be moved / removed to another computer but this is most secure. Important of course to backup the keepass biz.kdbx database file encrypted again in zip.
### Generate new key
How to generate a key and upload to digitalocean for using with droplet and saving to keepass
open puttygen
keep RSA change bits from 2048 to 4096
click generate
mouse around
change key comment to serversubdomainname.onayanova.com (or whatever the actual subdomain / domain is for this key)
save the private key temporarily to a folder (it will end up in keepass as an attachment)
DO NOT SET A PASSWORD HERE, keepass will secure the key, there's no need for the extra password and this is all about saving time
copy the public key box text and go to digitalocean Settings -> Security SSH keys section and add it as a new saved public key named same subdomina.onayanova.com or as appropriate
can close puttygen at this point if ever need it can get teh public key from the private key file again in future
Use this when creating a droplet, OR you can manually add it later via text editor: https://docs.digitalocean.com/products/droplets/how-to/add-ssh-keys/to-existing-droplet/
(note, the public key format displayed in puttygen when you open or generate the private key in the public key box is the only one that works with ssh, not the format used when it's saved to a file from puttygen )
### KeePass / KeeAgent
Make entry in biz keepass database and attach the private key to the entry then delete the file off disk once the keepass db file is saved.
How to automatically use keys from KeePass via KeeAgent plugin:
https://keeagent.readthedocs.io/en/stable/installation.html#windows
https://superuser.com/questions/905449/enable-ssh-keys-on-startup-in-keepass-with-keeagent-plugin
### Add key to server
Note that no matter how used the only valid public key is copied from the KeyGen public key textbox.
The public key file format if Saved from KeyGen is not useful for digitalocean or for ubunutu ssh
If generating a new droplet add public key to DigitalOcean by copying from KeyGen public key textbox
paste into new entry on d.o. in Settings -> Security SSH keys section and add it as a new saved public key named same subdomina.onayanova.com or as appropriate.
To manually add (or remove) a key from an existing server edit the following file:
`nano ./.ssh/authorized_keys`
copy from the public key field in PuttyGen and append to the above file.
Logout, confirm can login with that key then can remove any others not needed from above and elsewhere.
---
## Emailing from server
This is how to set up an ubuntu server to send emails out for system events and scripts.
It's actually pretty annoying in some cases and not necessarily helpful but here anyway.
apparmor _WILL_ conflict: https://linuxconfig.org/how-to-disable-apparmor-on-ubuntu-20-04-focal-fossa-linux and must exempt msmtp with these two commands:
ln -s /etc/apparmor.d/usr.bin.msmtp /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/disable/usr.bin.msmtp
## Installation
### Installation: apt-get install msmtp msmtp-mta -y
apt-get install msmtp msmtp-mta -y
### Config file for MSMTP with GMAIL
## Config file for MSMTP with GMAIL
```
```bash
###################################################
## Config file in /etc/msmtprc
##
## Set defaults for all accounts
### Config file in /etc/msmtprc
###
### Set defaults for all accounts
defaults
auth on
tls on
@@ -32,7 +322,7 @@ aliases /etc/msmtp_aliases
(note, need to set up 2fa in gmail first then create an app password which is what is used for the password here)
## Aliases file contents (/etc/msmtp_aliases):
### Aliases file contents (/etc/msmtp_aliases):
```
root: gzmailadmin@gmail.com
@@ -40,18 +330,18 @@ postmaster: gzmailadmin@gmail.com
default: gzmailadmin@gmail.com
```
## Testing
### Testing
`(echo "Subject: Test"; echo 'Body of the email') | msmtp cardjohn@ayanova.com`
## Email on boot or shutdown of server
### Email on boot or shutdown of server
Make this script in /etc/init.d/bootemail
```bash
#! /bin/sh
### BEGIN INIT INFO
#### BEGIN INIT INFO
# Provides: bootemail
@@ -67,7 +357,7 @@ Make this script in /etc/init.d/bootemail
# Description:
### END INIT INFO
#### END INIT INFO
PATH=/sbin:/usr/sbin:/bin:/usr/bin
@@ -99,192 +389,3 @@ return 0
Make it executable, from /etc/init.d folder execute: `chmod +x bootemail`
Enable: `update-rc.d bootemail defaults`
# System restart required info
https://linuxhandbook.com/system-restart-required-ubuntu/
# SSH
## Add existing user manually
(joeblogs is the user folder here, to create user need to use adduser add to sudo group etc, lots of guides online for that)
## Create the .ssh directory, and set its permissions
mkdir ~joebloggs/.ssh
chown joebloggs ~joebloggs/.ssh
chmod 0700 ~joebloggs/.ssh
## Create the authorized keys file, and set its permissions
nano ~joebloggs/.ssh/authorized_keys <--- Here, paste in the key from digitalocean keys in account settings security
chown joebloggs ~joebloggs/.ssh/authorized_keys
chmod 0700 ~joebloggs/.ssh/authorized_keys
# LETSENCRYPT
## Expanding / changing certificate
Changing a Certificate’s Domains
The --cert-name flag can also be used to modify the domains a certificate contains, by specifying new domains using the -d or --domains flag.
If certificate example.com previously contained example.com and www.example.com, it can be modified to only contain example.com
by specifying only example.com with the -d or --domains flag. Example:
certbot certonly --cert-name example.com -d example.com
The same format can be used to expand the set of domains a certificate contains, or to replace that set entirely:
certbot certonly --cert-name example.com -d example.org,www.example.org
SO FOR EXAMPLE for HelloAyaNova.com
certbot --nginx --cert-name helloayanova.com -d test.helloayanova.com,helloayanova.com,www.helloayanova.com,1665.helloayanova.com,7331.helloayanova.com,8888.helloayanova.com,a261.helloayanova.com
# NGINX
## Log
View dynamic errors and access logs using tail command:
tail -f /var/log/nginx/error.log
//view access log
tail -f /var/log/nginx/access.log
USEFUL COMMANDS
Get NGINX version from containerized nginx:
docker container exec -it \[container-id\] nginx -v
SUDO -> ROOT for session
sudo -i
(use exit to go back to john account)
Connect shell to running container:
docker exec -it \[container-id\] bash (or ash on alpine)
CERTBOT DRY RUN
append --dry-run to command
BACKUP FOLDERS AND FILES
cp -R pecklist pecklist_backup
REMOVE OLD .NET RUNTIMES:
Some of these commands get the job done
cd /usr/share/dotnet/shared
apt-cache pkgnames Microsoft\*
apt-cache pkgnames
apt list --installed
apt-get remove dotnet-runtime-3.0
# APT package manager
## Kept back package
If a package has been "kept back":
https://superuser.com/questions/1107334/apt-says-packages-have-been-kept-back-what-to-do/1108268
## Uninstall and purge package
sudo apt-get --purge remove package_name
apt-get autoremove
# SPACES BACKUP
Setup spaces utilities:
s3cmd: https://www.digitalocean.com/docs/spaces/resources/s3cmd/
manually copy a file to spaces:
```bash
/usr/bin/s3cmd put filename.zip s3://gztw1
```
# POSTGRES
## control service
sudo systemctl stop postgresql
sudo systemctl start postgresql
sudo systemctl restart postgresql
sudo systemctl status postgresql
## PSQL RUN SQL COMMANDS
Commands must be terminated with a semicolon or it will wait until it sees one
Open sql console as postgres admin user:
sudo -u postgres psql
### Quit psql
`\q`
### List all dbs
`\l`
### Get disk size of db
In bytes:
`select pg_database_size('AyaNova');`
Prettified to human readable sizes:
```sql
SELECT
pg_size_pretty (
pg_database_size ('AyaNova')
);
```
Display the size of _all_ databases in human readable format
```sql
SELECT
pg_database.datname,
pg_size_pretty(pg_database_size(pg_database.datname)) AS size
FROM pg_database;
```
# LOG FILES
show log file usage of system logs
```bash
root@devops:/var/ayanova/data/logs# du -h /var/log/
```
## Cleaning out log files
https://support.hostway.com/hc/en-us/articles/360001972270-How-to-clean-log-files-in-Linux
https://linuxhandbook.com/clear-systemd-journal-logs/
## Linux system journal logs cap
If down the road need to cap system journal logs size:
Add this to the standard server config script
cap system journal logs in linux to 250mb or some reasonable value
https://linuxhandbook.com/clear-systemd-journal-logs/
## Viewing huge log files
Use Less command, it has special commands to jump to start and end of file g G = go
q to quit
page up down and more if look online
# FIND BIG FILES
Find the biggest folders from any directory:
```bash
root@devops:/# du -h --max-depth=1 | sort -rh
```
Then go into each folder and run again to limit to that folder and subfolders
# DISK USAGE
```bash
df -h
```