Create pipeline transformation to simplify customizing log stream selectors

[kelnage]

Today, the only way to modify the Loki log stream selector (i.e., {app="grafana"}) when converting a Sigma rule is by either setting the custom attribute logsource_loki_selection to the selector you require in the rule (see this function), or by using the set_custom_attribute pipeline to achieve the same (see this class).

However, if we have one or more rules that contain a specific filter condition used in the query, we may want to promote those to the log stream selector, in order to improve query performance. A good example of this is Okta; in Grafana we currently ingest Okta log data into Loki with the eventType field as an label and many of the Sigma rules defined for Okta use the eventtype field as a top-level filter. In such a case, our pipeline would want to set a new value for logsource_loki_selection to look for the specific value that is being filtered in the Sigma rule.

To achieve this, I propose we create a new type of pipeline that allows the definition of log stream selectors based upon the value of fields in the detection and/or the logsource of the rule. As an example (although this syntax is not final, and is more than welcome to change), we might have a pipeline:

name: Okta
priority: 100
transformations:
    - id: select_loki_logs
      type: custom_log_stream_selector
      template: true
      selection:
          namespace: loki-ops
          name: $product-logs
          eventType|fieldref: eventtype

Given the following Sigma rule:

title: Okta Admin Role Assignment Created
...
logsource:
    product: okta
    service: okta
detection:
    selection:
        eventtype: 'iam.resourceset.bindings.add'
    condition: selection

And using the Loki backend should result in the following log stream selector being produced by the conversion process:

{namespace=`loki-ops`,name=`okta-logs`,eventType=`iam.resourceset.bindings.add`}

Scope/design considerations

• We will only want to include a condition if it is in the top level of the condition tree or it has a single negation in front of it - determining how to handle deeper conditions is out-of-scope for this task
• The logsource inclusion format $(category|product|service) is based upon the existing add_condition transformation pipeline template syntax, but different approaches could be used if well motivated
• Supporting regular expressions (|re) and negated detection selectors (not filter) should be relatively simple from a query perspective, given they are directly supported by Loki
• If we have list of values for the (e.g., such as used in this rule) are provided, they could be converted into a regular expression match (akin to how SigmaString conditions are converted into unbound line filters) - but this is not a necessary requirement - if it is not implemented, such a field should be ignored for the purpose of conversion
• The interaction between this pipeline and set_custom_attribute transformation could be complex - and handling this could be done in multiple ways - but for the time being, I would suggest this should be out of scope and each should simply overwrite any existing value of the logsource_loki_selection if there is one