admin/raven
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Browse Source
This commit is contained in:
John Cardinal
2022-09-10 23:03:00 +00:00
parent 9aea219bb6
commit 98461fb0dd
1 changed files with 14 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
devdocs/todo.txt
View File

@@ -8,19 +8,26 @@ Problem:
AyaNova log and system journal eating up all disk space due to postgres being down for unknown reason and ayanova repeatedly trying to do stuff in this condition and then issuing errors and LOGGING AyaNova log and system journal eating up all disk space due to postgres being down for unknown reason and ayanova repeatedly trying to do stuff in this condition and then issuing errors and LOGGING
potential fixes: TODO HERE: potential fixes: TODO HERE:
Detect postgres is unreachable and stop digging a hole, wait to resolve it before continuing on again FIXED Detect postgres is unreachable and stop digging a hole, wait to resolve it before continuing on again
Don't log same message repeatedly
actually this is not necessary if just stop running jobs when can't reach sql server
FIXED don't dual log to system journal logs and ayanova log FIXED don't dual log to system journal logs and ayanova log
Ayanova is not only logging but also directing it's output to the terminal window which in linux system service is being logged also to the journal logs Ayanova is not only logging but also directing it's output to the terminal window which in linux system service is being logged also to the journal logs
AyaNova should only ouput to it's log file, not to terminal with the sole exception of the initial boot up period AyaNova should only ouput to it's log file, not to terminal with the sole exception of the initial boot up period
This may be a configuration in the logging library set this way?? This may be a configuration in the logging library set this way??
cap system journal logs in linux TODO: Add this to the standard server config script
cap system journal logs in linux to 250mb or some reasonable value
https://linuxhandbook.com/clear-systemd-journal-logs/ https://linuxhandbook.com/clear-systemd-journal-logs/
cap ayanova logs in configuration TODO: cap ayanova logs in configuration
If detect db is unavailable then log that and stop job processor etc basically freeze the server jobs until the db becomes available something in here, but maybe it's config is different compare with code for logrotate in use now and see what the exact settings are
https://github.com/NLog/NLog/wiki/Configuration-file
https://github.com/NLog/NLog/wiki/File-target#archival-options
https://nlog-project.org/config/
NOTE: it may make sense to swtich from rotating every week to rotating on size instead, after all why force to rotate a nearly empty log?
DONE: If detect db is unavailable then log that and stop job processor etc basically freeze the server jobs until the db becomes available
server state db unavailble not just at boot but during ops server state db unavailble not just at boot but during ops
quick fixes: quick fixes:
https://support.hostway.com/hc/en-us/articles/360001972270-How-to-clean-log-files-in-Linux https://support.hostway.com/hc/en-us/articles/360001972270-How-to-clean-log-files-in-Linux
https://linuxhandbook.com/clear-systemd-journal-logs/ https://linuxhandbook.com/clear-systemd-journal-logs/