fluentdForward https breaks in 4.6.0-202103202154.p0

[adrianchifor]

Describe the bug
We've been using https:// for the fluentdForward output urls in the ClusterLogForwarder and it was fine until 4.6.0-202103202154.p0. The upgrade this morning broke our logging and fluentd is not longer forwarding anything, unless we change the protocol to tcp://.

Additionally, we've noticed that the fluentd secret in openshift-logging has empty ca-bundle.crt, tls.crt and tls.key while before these were populated (in 4.6.0-202011221454.p0); master-certs secret is missing as well.

I have a feeling this PR #823 is responsible for the issue.

Environment

• OCP 4.6.21
• cluster-logging operator 4.6.0-202103202154.p0

Logs
Nothing relevant in the operator, but when using https:// in fluentdForward url, in fluentd we see

2021-04-03 15:59:26 +0000 [warn]: TLS transport is configured in insecure way

likely because of the missing contents in fluentd secret and missing master-certs.

Expected behavior
Operator should correctly evaluate https:// fluentdForward urls and populate the contents of the fluentd secret with the ca-bundle, tls cert and key, and create the master-certs secret as well, to tell fluentd to use TLS.

Actual behavior
Operator fails to populate fluentd secret in openshift-logging and create master-certs secret. These cause fluentd log forwarding to be misconfigured.

To Reproduce
Setup ClusterLogForwarder to forward logs to logstash:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: logstash
      type: fluentdForward
      url: https://<logstash URL>:16221  # used to work before, now only tcp:// works
  pipelines:
    - name: container-logs
      inputRefs:
        - application
      outputRefs:
      - logstash
    - name: infra-logs
      inputRefs:
        - infrastructure
      outputRefs:
      - logstash
    - name: audit-logs
      inputRefs:
        - audit
      outputRefs:
      - logstash

and ClusterLogging

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  managementState: Managed
  collection:
    logs:
      type: fluentd

[jcantrill]

please open an issue at issues.redhat.com if you are still experiencing a problem. A number of the certificate issues likely were addressed by various bugs.