DopplerHQ/kubernetes-operator

GCP GKE INFO logs are showing ERROR

jheyer159 opened this issue · 5 comments

jheyer159 commented

Followed the README to install the operator using kubectl and it seems to be running as normal, but every minute I'm getting a chunk of error logs that show INFO in the message and are showing as ERROR.

Is there something I need to change to correct the output?

Screen Shot 2023-03-10 at 9 30 22 AM

...
logName: "projects/project_id/logs/stderr",
severity: "ERROR"
textPayload: "2023-03-10T14:26:47.498Z	INFO	..."
...
nmanoogian commented

Hey @jheyer159! Thanks for writing in about this. The operator uses the go-logr to write those messages and it does look like they're being sent with log.Info which should end up on stdout.

Let me do a bit of digging and see if I'm able to reproduce this.

  • 👀1
mrasnake-doppler commented

Hello @jheyer159, I wanted to update you as we were able to determine that logs sent with log.Info are sent to stderr which is likely why they are being categorized with severity: "ERROR".

However, this is the default setting and not something we are directly defining. So we are going to to do some research into the appropriate conventions for similar operators before deciding to make a change or not.

  • 👍1
jheyer159 commented

I tried applying this patch using kustomize

kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: doppler-operator-controller-manager
  namespace: doppler-operator-system
spec:
  template:
    spec:
      containers:
        - name: manager
          env:
            - name: ZAP_OPTIONS
              value: |
                {
                  "level": "info",
                  "outputPaths": ["stdout"],
                  "errorOutputPaths": ["stderr"],
                  "encoding": "console"
                }

it seems to be active, but i'm still getting error logs. Any idea if this should work? I'm not sure how to debug it further.

nmanoogian commented

@jheyer159 That could be an interesting solution! Where did you see the ZAP_OPTIONS environment variable? I couldn't find it in the zap or controller-runtime docs.

jheyer159 commented

You're right - that was an incorrect suggestion from GPT. It seems the CLI options are found here and there's not one for changing DestWriter

https://github.com/kubernetes-sigs/controller-runtime/blob/0ef07535542bb44dc642fe5c67dcd4f7a9c06a22/pkg/log/zap/zap.go#L263

I will explore changing this in a fork.

  • 👍1