signalfx/splunk-otel-collector

config is evaluated twice. having to double up on `$`

rockb1017 opened this issue · 6 comments

rockb1017 commented

when using filelog receiver, $ is used for accessing data from entry.
https://github.com/open-telemetry/opentelemetry-log-collection/blob/main/docs/types/field.md
and i have an example config here.

  filelog:
    encoding: nop
    exclude:
    - /var/log/pods/default_rock-sck-otel*_*/otelcollector/*.log
    - /var/log/pods/kube-system_*/*/*.log
    fingerprint_size: 1kb
    include:
    - /var/log/pods/*/*/*.log
    include_file_name: false
    include_file_path: true
    max_concurrent_files: 1024
    max_log_size: 1MiB
    operators:
    - id: get-format
      routes:
      - expr: $$body matches "^\\{"
        output: parser-docker
      - expr: $$body matches "^[^ Z]+ "
        output: parser-crio
      - expr: $$body matches "^[^ Z]+Z"
        output: parser-containerd
      type: router
    - id: parser-crio
      regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$
      timestamp:
        layout: "2006-01-02T15:04:05.000000000-07:00"
        layout_type: gotime
        parse_from: time
      type: regex_parser
    - combine_field: log
      id: crio-recombine
      is_last_entry: ($.logtag) == 'F'
      type: recombine
    - id: check for empty log
      ops:
      - add:
          field: log
          if: EXPR($.log) != nil
          value: nil
      output: filename
      type: restructure
    - id: parser-containerd
      regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) (?P<log>.*)$
      timestamp:
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        parse_from: time
      type: regex_parser
    - combine_field: log
      id: containerd-recombine
      is_last_entry: ($.logtag) == 'F'
      output: filename
      type: recombine
    - id: parser-docker
      timestamp:
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        parse_from: time
      type: json_parser
    - id: filename
      resource:
        service.name: EXPR($$attributes["file.path"])
      type: metadata
    - id: extract_metadata_from_filepath
      parse_from: $$attributes["file.path"]
      regex: ^\/var\/log\/pods\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[^\/]+)\/(?P<container_name>[^\._]+)\/(?P<run_id>\d+)\.log$
      type: regex_parser
    - resource:
        com.splunk.sourcetype: EXPR("kube:container:"+$.container_name)
        container_name: EXPR($.container_name)
        k8s.namespace.name: EXPR($.namespace)
        k8s.pod.name: EXPR($.pod_name)
        k8s.pod.uid: EXPR($.uid)
        run_id: EXPR($.run_id)
        stream: EXPR($.stream)
      type: metadata
    - id: clean-up-log-record
      ops:
      - move:
          from: log
          to: $
      type: restructure
    poll_interval: 200ms
    resource:
      test-scenaro: 8198:200
    start_at: beginning

but the $$body is parsed and changed to env var value of body. So we can't use image from this repo for unified helm chart. I think some changes are required in the cmd/otelcol/main.go

dmitryax commented

It seems like we have a bug somewhere in https://github.com/signalfx/splunk-otel-collector/tree/main/internal/configprovider . core/contrib works well

dmitryax commented

Did some initial investigation.

Looks like config is evaluated twice in our distro.

Any $$key is evaluated to $key by configprovider.Manager.Resolve first (and shown correctly in http://localhost:55554/debug/configz/effective), but then components get config with an empty value or any other value set for $key env var.

rockb1017 commented

Thanks @dmitryax from what you shared, I tried something and it worked. I doubled up $. and everything worked as it should.

rockb1017 commented

should we fix it or just use $$$$ in the helm chart?

dmitryax commented

should we fix it or just use $$$$ in the helm chart?

we can do that as an ugly short term workaround to unblock moving to our image. I'll try to fix this bug in the meantime

  • 👍1