opensearch-project/data-prepper

[BUG] EVENT logging masks all inputs

dlvenable opened this issue · 0 comments

The EVENT mask is intended to be a signal to the logger to mask only Data Prepper Event objects. However, it currently operates like a SENSITIVE marker which should mask all fields (except exceptions see #3375).

For example, add_entries has this exception.

LOG.error(EVENT, "Error adding entry to record [{}] with key [{}], metadataKey [{}], value_expression [{}] format [{}], value [{}]",
recordEvent, entry.getKey(), entry.getMetadataKey(), entry.getValueExpression(), entry.getFormat(), entry.getValue(), e);

It results in logs like the following:

Error adding entry to record [******] with key [******], metadataKey [******], value_expression [******] format [******], value [******] java.lang.ClassCastException: null

Solution:

Fix the SensitiveArgumentMaskingConverter to only mask Event objects when the marker is EVENT rather than SENSITIVE.

Also, we should audit our usages of EVENT. For each usage, we should consider 1) Can we keep it as EVENT as is? 2) Do we need to remove some fields to keep as EVENT; or 3) Do we need to make it SENSITIVE?

Tasks

  • Fix the SensitiveArgumentMaskingConverter
  • Audit and update usages of the DataPrepperMarkers.EVENT marker