Add memory guard-rails
Closed this issue · 0 comments
baubakg commented
We need two types of guard rails.
- Global memory guard rails.
- File level data guard rails
File level guard rails
If a file has too many lines we stop at an upper limit of entries.
See store the skipped/cut files in a shared location
We have the following properties:
- PROP_LOGPARSER_FILEENTRY_LIMIT
- PROP_LOGPARSER_FILESIZE_LIMIT
- PROP_LOGPARSER_HEAP_LIMIT
- PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE
- PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT
| Property | Description | Affects | Scale | Default Value |
|---|---|---|---|---|
| PROP_LOGPARSER_FILEENTRY_LIMIT | Maximum number of entries to parse per file | File parsing | Count | -1 (disabled) |
| PROP_LOGPARSER_FILESIZE_LIMIT | Maximum file size in MB to parse | File parsing | Megabytes | -1 (disabled) |
| PROP_LOGPARSER_HEAP_LIMIT | Maximum heap size increase in MB before warning | File parsing, FilterBy, Search, enrich, groupBy | Megabytes | -1 (disabled) |
| PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE | Maximum percentage of memory usage before warning | File parsing, FilterBy, Search, enrich, groupBy | Percentage | -1 (disabled) |
| PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT | Whether to throw exception when memory limits exceeded | Error handling | Boolean | false |