admin/raven
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Browse Source
This commit is contained in:
John Cardinal
2021-12-23 15:15:14 +00:00
parent 7d357270f6
commit e754230cbd
9 changed files with 42 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/8.0/ayanova/docs/ops-config-environment-variables.md
View File

@@ -85,7 +85,6 @@ Command line parameters are appended to the command to start the ayanova server,
- [AYANOVA_REPORT_RENDER_BROWSER_PATH](ops-config-report-render-browser-path.md)
- [AYANOVA_REPORT_RENDERING_TIMEOUT](ops-config-report-rendering-timeout.md)
- [AYANOVA_REPORT_RENDERING_MAX_INSTANCES](ops-config-report-rendering-max-instances.md)
### SECURITY

64
docs/8.0/ayanova/docs/ops-config-report-rendering-max-instances.md
View File

@@ -1,64 +0,0 @@
# REPORT RENDERING MAXIMUM INSTANCES SETTING
The report rendering maximum instances value controls how many report rendering processors are allowed to be running when a report is submitted before AyaNova will start forcibly shutting down excess instances.
Report rendering is a relatively "expensive" process in terms of memory and cpu usage on the server taking far more resources than non reporting AyaNova operations and in some rare cases the report rendering process can get stuck in a loop or frozen due to a bug in a report template script or simply selecting too many records to report in a practical amount of time. If too much memory or CPU cycles are tied up by these "zombie" report rendering processes it can cause the AyaNova server to stop responding completely.
When AyaNova receives a report request it checks first to ensure there are not more report rendering processes active than are specified in this setting. If there are too many already running it will return a busy code to the end user to try again later and will attempt to forcibly shut down any of the processes that should have expired. Expired here means processes that have been running longer than [AYANOVA_REPORT_RENDERING_TIMEOUT](ops-config-report-rendering-timeout.md) number of milliseconds.
It's important to note that this setting does not prevent AyaNova from processing a large time consuming report for any length of time necessary to complete it; this setting only takes effect when another report is requested to be rendered and there are no slots free for report rendering. For this reason, if a User has to render an unusually large report that is known to take longer than the timeout period they would be advised to render that report when no one else is requesting a report (i.e. after hours).
## Performance implications
Note that increasing report rendering settings too high can result in the server running out of physical memory or CPU capacity. When increasing the limits be mindful of this potential and monitor the server carefully to ensure it's not "bottlenecking". The reporting capacity scales directly with the amount of memory and CPU capacity available to the server.
## Default
If no override is specified AyaNova will use the following default value:
`3`
This means AyaNova can normally be processing 3 report requests simultaneously and is generally an adequate setting for most situations as most reports are generated in a few seconds freeing up a slot immediately after they are completed.
## When to change this setting
If users *regularly* receive error messages that the server is too busy when rendering reports it may indicate the need to increase this value. As report rendering is an intensive operation for the server consuming a great deal of memory and cpu cycles it is best to only raise this value 1 at a time and then see if that resolves the issue.
## MINIMUM / MAXIMUM
There is a limit of 10 instances maximum built into AyaNova so specifying a value greater than 10 will be ignored and 10 will be used instead.
Specifying a value less than 1 will be automatically changed to 1 as the minimum allowed value.
## Overriding
AyaNova expects this value to be provided by a config.json property, environment variable or command line parameter named
`AYANOVA_REPORT_RENDERING_MAX_INSTANCES`
The value specified should be a string specifying the value as an integer, for example:
`5`
Example config.json entry
```json
{
...other properties...
"AYANOVA_REPORT_RENDERING_MAX_INSTANCES": "5"
}
```
Example command line parameter
`ayanova.exe --AYANOVA_REPORT_RENDERING_MAX_INSTANCES="5`
Example environment variable
Windows
`set "AYANOVA_REPORT_RENDERING_MAX_INSTANCES=5"`
Linux / MAC
`export AYANOVA_REPORT_RENDERING_MAX_INSTANCES="5"`

27
docs/8.0/ayanova/docs/ops-config-report-rendering-timeout.md
View File

@@ -1,8 +1,8 @@
# REPORT RENDERING TIMEOUT SETTING
The report rendering timeout value controls how long AyaNova will wait for a prior report to complete rendering when it receives a request to render a subsequent report.
The report rendering timeout value controls how long AyaNova will wait for a report to complete rendering.
This avoids a situation where a subsequent user can not get their report because a prior report generation is frozen due to a bug in a report template script or simply selecting too many records to report in a practical amount of time.
This avoids a situation where a report generation is frozen due to a bug in a report template script or simply selecting too many records to report in a practical amount of time.
## Performance implications
@@ -10,24 +10,23 @@ Note that increasing report rendering settings too high can result in the server
## How timeouts work
When a report is requested to be rendered AyaNova starts the process and makes note of the time and will process that report indefinitely for as long as it takes to complete. However, if a subsequent report request is made and the first report has not completed yet then AyaNova will use this setting to wait for the first report to process before starting the next one. If the timeout period completes without the first report completed rendering then AyaNova will forcibly stop processing the first report and release the resources dedicated to it and start processing the new report.
When a report is requested to be rendered AyaNova starts the process and makes note of the time and will process that report until completed or the timeout is reached.
It's important to note that this timeout does not prevent AyaNova from processing a report for any length of time necessary to complete it; this timeout only takes effect if another report is requested to be rendered so if a User has to render a very large report that is known to take longer than the timeout period they would be advised to render that report when no one else is requesting a report (i.e. after hours)
The value is specified in **milliseconds** or thousandths of a second. For example a 30 second timeout is 30000 milliseconds.
The value is specified in **minutes**.
## Default
If no override is specified AyaNova will use the following default value:
`30000`
`3`
This means AyaNova will wait for a prior report to complete for no longer than 30,000 milliseconds or 30 seconds before it will stop that prior report render and start the new one.
This means AyaNova will wait for a prior report to complete for no longer than 3 minutes before it will stop and return an error.
## MINIMUM / MAXIMUM
There is a hard cap of 3 minutes or 180000 milliseconds built into AyaNova so specifying a value greater than 180000 milliseconds will be ignored and 3 minutes used instead.
Specifying a value less than 1 second (1000) will be automatically changed to 1000 as the minimum allowed value.
There is a hard cap of 15 minutes built into AyaNova so specifying a value greater than 15 minutes will be ignored and 15 minutes used instead.
Specifying a value less than 1 minute will be automatically changed to 1 minute as the minimum allowed value.
## Overriding
@@ -35,7 +34,7 @@ AyaNova expects this value to be provided by a config.json property, environment
`AYANOVA_REPORT_RENDERING_TIMEOUT`
The value specified should be a string specifying the timeout in **milliseconds**, for example:
The value specified should be a string specifying the timeout in **minutes**, for example:
`30000`
Example config.json entry
@@ -43,20 +42,20 @@ Example config.json entry
```json
{
...other properties...
"AYANOVA_REPORT_RENDERING_TIMEOUT": "45000"
"AYANOVA_REPORT_RENDERING_TIMEOUT": "4"
}
```
Example command line parameter
`ayanova.exe --AYANOVA_REPORT_RENDERING_TIMEOUT="30000`
`ayanova.exe --AYANOVA_REPORT_RENDERING_TIMEOUT="5`
Example environment variable
Windows
`set "AYANOVA_REPORT_RENDERING_TIMEOUT=60000"`
`set "AYANOVA_REPORT_RENDERING_TIMEOUT=4"`
Linux / MAC
`export AYANOVA_REPORT_RENDERING_TIMEOUT="120000"`
`export AYANOVA_REPORT_RENDERING_TIMEOUT="12"`

2
docs/8.0/ayanova/docs/ops-config-set-superuser-pw.md
View File

@@ -22,7 +22,7 @@ Example config.json entry
```json
{
...other properties...
"AYANOVA_REPORT_RENDERING_TIMEOUT": "Th3RainInSpainFallsMainlyOnTh3Pla1n"
"AYANOVA_SET_SUPERUSER_PW": "Th3RainInSpainFallsMainlyOnTh3Pla1n"
}
```

1
docs/8.0/ayanova/mkdocs.yml
View File

@@ -158,7 +158,6 @@ nav:
- 'PORT and URL configuration': 'ops-config-use-urls.md'
- 'Reporting render browser path': 'ops-config-report-render-browser-path.md'
- 'Reporting timeout configuration': 'ops-config-report-rendering-timeout.md'
- 'Reporting max instances configuration': 'ops-config-report-rendering-max-instances.md'
- 'Environment variable reference': 'ops-config-environment-variables.md'
- 'Security and JWT web tokens': 'ops-config-jwt-secret.md'
- 'SuperUser password reset': 'ops-config-set-superuser-pw.md'

10
server/AyaNova/Controllers/ReportController.cs
View File

@@ -196,12 +196,8 @@ namespace AyaNova.Api.Controllers
if (!serverState.IsOpen)
return StatusCode(503, new ApiErrorResponse(serverState.ApiErrorCode, null, serverState.Reason));
//return if no free slot
//Note that this *should* normally return a 503 however this is not an unexpected result and we're pretty tightly wired into that meaning the server is closed at the client end which
//handles it at a lower api level
//returning an OK method here allows the client to handle it at the level of the report dialog rather than the api handler which will short circuit if it was a 503
if (!Util.ReportRenderManager.RenderSlotAvailable(log))
return Ok(ApiOkResponse.Response(new { busy = true, retryms = ServerBootConfig.AYANOVA_REPORT_RENDERING_TIMEOUT }));
//check for an kill any expired prior renders stuck around
Util.ReportRenderManager.KillExpiredRenders(log);
ReportBiz biz = ReportBiz.GetBiz(ct, HttpContext);
if (!Authorized.HasReadFullRole(HttpContext.Items, biz.BizType))
@@ -213,9 +209,9 @@ namespace AyaNova.Api.Controllers
var API_URL = $"http://127.0.0.1:{httpConnectionFeature.LocalPort}/api/v8/";
try
{
var Expires = DateTime.UtcNow.AddMinutes(ServerBootConfig.AYANOVA_REPORT_RENDERING_TIMEOUT);
var result = await biz.RenderReport(reportRequest, API_URL);
if (string.IsNullOrWhiteSpace(result))
return BadRequest(new ApiErrorResponse(biz.Errors));
else

4
server/AyaNova/biz/ReportBiz.cs
View File

@@ -434,9 +434,7 @@ namespace AyaNova.Biz
}
//Default timeout for each operation of report generation
// var WaitTimeout = new WaitForFunctionOptions() { Timeout = ServerBootConfig.AYANOVA_REPORT_RENDERING_TIMEOUT*10 };
//includeWoItemDescendants?
reportRequest.IncludeWoItemDescendants = report.IncludeWoItemDescendants;

59
server/AyaNova/util/ReportProcessManager.cs
View File

@@ -6,20 +6,10 @@ using Microsoft.Extensions.Logging;
namespace AyaNova.Util
{
/// <summary>
/// Used by reporting system to ensure headless browsers don't hang around in an untimely manner chewing up resources
/// needed due to bugs in puppeteersharp where it won't close the browser on timeout properly
/// also zombie process issues in linux etc, this just ensures it's safe
/// This is triggered when a report is rendered on demand
/// in other words demand drives whether it kills long running renders or not
/// this is by design to allow a scenario where a super long running report can still be run off hours (for example)
/// </summary>
//Track processes and kill any that go past their expiry date
internal static class ReportRenderManager
{
/*
expired processes are removed by the act of trying to get a new slot so in this way it still supports running super long reports overnight for example as long as there is no contention
The other way was by a job that looks for expired processes but that would mean all old jobs would expire all the time so there would be an issue with huge reports never working
*/
//thread safe collection for unordered items, optimized for single thread produce/consume (which is the norm here) but supports multithread produce / consume (which is needed for separate cleanup job)
internal static ConcurrentBag<ReportRenderInstanceInfo> _baginstances;// = new ConcurrentBag<ReportRenderInstanceInfo>();
@@ -34,45 +24,35 @@ namespace AyaNova.Util
internal int ReporterProcessId { get; set; }
internal DateTime Expires { get; set; }
internal ReportRenderInstanceInfo(int processId)
internal ReportRenderInstanceInfo(int processId, DateTime expires)
{
ReporterProcessId = processId;
Expires = DateTime.UtcNow.AddMilliseconds(ServerBootConfig.AYANOVA_REPORT_RENDERING_TIMEOUT);
Expires = expires;
}
}
internal static bool RenderSlotAvailable(ILogger log)
internal static void KillExpiredRenders(ILogger log)
{
log.LogTrace("RenderSlotAvailable check");
//var count = _baginstances.Count;
// #if (DEBUG)
// log.LogInformation($"DBG: RenderSlotAvailable check, there are currently {_baginstances.Count} instances in the bag");
// #endif
if (_baginstances.Count >= ServerBootConfig.AYANOVA_REPORT_RENDERING_MAX_INSTANCES)
log.LogTrace("Clear expired render check");
//check for expired and remove
var Instances = _baginstances.ToArray();
var dtNow = DateTime.UtcNow;
foreach (ReportRenderInstanceInfo i in Instances)
{
log.LogTrace($"RenderSlotAvailable there are no free report rendering slots available, current count is {_baginstances.Count}, checking for expired slots to force closed");
//check for expired and remove
var Instances = _baginstances.ToArray();
var dtNow = DateTime.UtcNow;
foreach (ReportRenderInstanceInfo i in Instances)
if (i.Expires < dtNow)
{
if (i.Expires < dtNow)
{
// #if (DEBUG)
// log.LogInformation($"DBG: RenderSlotAvailable attempting kill of expired process {i.ReporterProcessId}");
// #endif
ForceCloseProcess(i, log);
}
#if (DEBUG)
log.LogInformation($"DBG: KillExpiredRenders attempting kill of expired process {i.ReporterProcessId}");
#endif
ForceCloseProcess(i, log);
}
}
//allow to continue if there are now fewer than max instances in the bag
return _baginstances.Count < ServerBootConfig.AYANOVA_REPORT_RENDERING_MAX_INSTANCES;
}
internal static bool ForceCloseProcess(ReportRenderInstanceInfo instance, ILogger log)
{
log.LogTrace($"ForceCloseProcess on instance id {instance.ReporterProcessId} started {instance.Expires.ToString()} utc");
log.LogTrace($"ForceCloseProcess on report render instance id {instance.ReporterProcessId} exired {instance.Expires.ToString()} utc");
try
{
var p = Process.GetProcessById(instance.ReporterProcessId);
@@ -83,8 +63,7 @@ namespace AyaNova.Util
p.Kill();
if (p.HasExited == false)
{
log.LogDebug($"RenderSlotAvailable oldest slot could not be stopped");
log.LogDebug($"Expired report render instance id {instance.ReporterProcessId} could not be force closed");
return false;//can't kill it so can't free up a slot
}
}
@@ -105,12 +84,12 @@ namespace AyaNova.Util
}
}
internal static void AddProcess(int processId, ILogger log)
internal static void AddProcess(int processId, DateTime expires, ILogger log)
{
// #if (DEBUG)
// log.LogInformation($"DBG: RenderSlotAvailable::AddProcess {processId} in the bag");
// #endif
_baginstances.Add(new ReportRenderInstanceInfo(processId));
_baginstances.Add(new ReportRenderInstanceInfo(processId, expires));
// #if (DEBUG)
// log.LogInformation($"DBG: RenderSlotAvailable::AddProcess, there are currently {_baginstances.Count} instances in the bag");

14
server/AyaNova/util/ServerBootConfig.cs
View File

@@ -61,7 +61,7 @@ namespace AyaNova.Util
internal static string AYANOVA_JWT_SECRET { get; set; }
internal static string AYANOVA_USE_URLS { get; set; }
internal static int AYANOVA_REPORT_RENDERING_TIMEOUT { get; set; }
internal static int AYANOVA_REPORT_RENDERING_MAX_INSTANCES { get; set; }
//DATABASE
@@ -181,15 +181,11 @@ namespace AyaNova.Util
//PROCESS CONTROL
int? nTemp = config.GetValue<int?>("AYANOVA_REPORT_RENDERING_TIMEOUT");
AYANOVA_REPORT_RENDERING_TIMEOUT = (null == nTemp) ? 30000 : (int)nTemp;//default is 30 seconds
if (AYANOVA_REPORT_RENDERING_TIMEOUT < 1000) AYANOVA_REPORT_RENDERING_TIMEOUT = 1000; //one second minimum timeout
if (AYANOVA_REPORT_RENDERING_TIMEOUT > 180000) AYANOVA_REPORT_RENDERING_TIMEOUT = 180000; //3 minutes maximum timeout
nTemp = config.GetValue<int?>("AYANOVA_REPORT_RENDERING_MAX_INSTANCES");
AYANOVA_REPORT_RENDERING_MAX_INSTANCES = (null == nTemp) ? 3 : (int)nTemp;
if (AYANOVA_REPORT_RENDERING_MAX_INSTANCES < 1) AYANOVA_REPORT_RENDERING_MAX_INSTANCES = 1; //minimum instances
if (AYANOVA_REPORT_RENDERING_MAX_INSTANCES > 10) AYANOVA_REPORT_RENDERING_MAX_INSTANCES = 10; //Fixed maximum instances
AYANOVA_REPORT_RENDERING_TIMEOUT = (null == nTemp) ? 3 : (int)nTemp;//default is 3 minutes
if (AYANOVA_REPORT_RENDERING_TIMEOUT < 1) AYANOVA_REPORT_RENDERING_TIMEOUT = 1; //one minute minimum timeout
if (AYANOVA_REPORT_RENDERING_TIMEOUT > 15) AYANOVA_REPORT_RENDERING_TIMEOUT = 15; //15 minutes maximum timeout
//DB
AYANOVA_DB_CONNECTION = config.GetValue<string>("AYANOVA_DB_CONNECTION");
bTemp = config.GetValue<bool?>("AYANOVA_PERMANENTLY_ERASE_DATABASE");