adobe/log-parser

Add memory guard-rails

Closed this issue · 0 comments

We need two types of guard rails.

  • Global memory guard rails.
  • File level data guard rails

File level guard rails

If a file has too many lines we stop at an upper limit of entries.

See store the skipped/cut files in a shared location

We have the following properties:

  • PROP_LOGPARSER_FILEENTRY_LIMIT
  • PROP_LOGPARSER_FILESIZE_LIMIT
  • PROP_LOGPARSER_HEAP_LIMIT
  • PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE
  • PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT
Property Description Affects Scale Default Value
PROP_LOGPARSER_FILEENTRY_LIMIT Maximum number of entries to parse per file File parsing Count -1 (disabled)
PROP_LOGPARSER_FILESIZE_LIMIT Maximum file size in MB to parse File parsing Megabytes -1 (disabled)
PROP_LOGPARSER_HEAP_LIMIT Maximum heap size increase in MB before warning File parsing, FilterBy, Search, enrich, groupBy Megabytes -1 (disabled)
PROP_LOGPARSER_MEMORY_LIMIT_PERCENTAGE Maximum percentage of memory usage before warning File parsing, FilterBy, Search, enrich, groupBy Percentage -1 (disabled)
PROP_LOGPARSER_EXCEPTION_ON_MEMORY_LIMIT Whether to throw exception when memory limits exceeded Error handling Boolean false