Labels from ClusterLogForwarder are prefixed by openshift.labels.

[Avarei]

Describe the bug
Labels from ClusterLogForwarder are prefixed by openshift.labels.

Environment

• OpenShift 4.7.13 (Bare Metal)
• Red Hat OpenShift Logging 5.0.5-11

Expected behavior
Logs contain a foo: bar field

Actual behavior
Logs contain a openshift.labels.foo: bar field

To Reproduce
Steps to reproduce the behavior:

1. Create ClusterLogging instance using EFK stack like described in the OpenShift Documentation
2. create a Clusterlogforwarder with a label:

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance 
  namespace: openshift-logging 
spec:
  outputs: []
  pipelines:
   - name: enable-default-log-store 
     inputRefs:
     - application
     - infrastructure
     - audit
     outputRefs:
     - default 
     labels:
       foo: bar

3. Wait for the FluentD Pods to reboot
4. View the Kibana Dashboard Logs

[jcantrill]

This is by design: https://github.com/openshift/enhancements/blob/master/enhancements/cluster-logging/forwarder-tagging.md These are static labels that are added to every message.

[Avarei]

This is unexpected and undocumented behaviour (undocumented in the OpenShift documentation and in the ClusterLogForward CRD)

In the Design document you sent, I do not see any reasoning for this kind of prefix, nor can I see any benefit of having it.

This Design decision introduces a Limitation which is seemingly without payoff, since it would also be possible to set openshift.labels.foo: bar as a label if desired.
As the key and value are set by the Admin I believe the Admin should be responsible for choosing its name.

If an Admin aggregates the logs at a central log stack from VMs, applications AND OpenShift Clusters they may want to add these kind of static labels to all of their logs. (For example for classification of logs)

I ask you to reconsider the decision of prepending this prefix.

The alternatives that I see are far worse, as admins would be forced to either:

• rewrite the log entry at the central log stack
• run this operator in the unmanaged Mode, to overwrite the fluentd config.

I would like to reopen this issue for discussion.

[jcantrill]

We do not allow an arbitrary envelope on the output side but have one that has been well defined for 10+ releases. The output is a JSON document. Your bug doesn’t state your expectation nor does it consider what we have historically provided. Given this is now part of a GA product for which we did the due diligence of following the Openshift enhancement model, I don’t see how this will change, but I will defer to @alanconway if he believes otherwise.

On Mon, Jul 26, 2021 at 2:42 PM Tim ***@***.***> wrote: This is unexpected and undocumented behaviour (undocumented in the OpenShift documentation and in the ClusterLogForward CRD

If this is true then please open a bug at issues.redhat.com for resolution In the Design document you sent, I do not see any reasoning for this kind

of prefix, nor can I see any benefit of having it

The reason is we desire to control the output envelope because there are an unknown number of containers logging in arbitrary formats that we are attempting to normalize to a single format This Design decision introduces a Limitation which is seemingly without

payoff, since it would also be possible to set openshift.labels.foo: bar as a label if desired. As the key and value are set by the Admin I believe the Admin should be responsible for choosing its name. If an Admin aggregates the logs at a central log stack from VMs, applications AND OpenShift Clusters they may want to add these kind of static labels to all of their logs. (For example for classification of logs

This feature was specifically added for one customer to do exactly what you suggest, specifically for them to add something like the cluster name

I ask you to reconsider the decision of prepending this prefix. The alternatives that I see are far worse, as admins would be forced to either: - rewrite the log entry at the central log stack - run this operator in the unmanaged Mode, to overwrite the fluentd config. I am missing exactly what you see are the limitations. This allows you to

define any static label you require on any openshift cluster. You can push these messages to any output we support. This only is opinionated about the root where the key value lives. You simply need to account for that root when you query

- I would like to reopen this issue for discussion. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1117 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCWOOFQL2FSEM7G5ZTWUPLTZWUBXANCNFSM5A7XD2TA> .

-- -- Jeff Cantrill Principal Software Engineer, Red Hat Engineering OpenShift Logging Red Hat, Inc. ***@***.*** http://www.redhat.com