Community operator pod is crashing

Question

Community operator pod is crashing

cmoulliard opened this issue 5 years ago · 2 comments

Issue

The community operator pod is crashing and reports the following error invalid source, secret specified is malformed - upstream-community-operators on okd 3.11

time="2019-04-04T07:10:02Z" level=info msg="Using in-cluster kube client config" port=50051 type=appregistry
--
  | time="2019-04-04T07:10:02Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr\|community-operators  --registry=https://quay.io/cnr\|upstream-community-operators ]" port=50051 type=appregistry
  | time="2019-04-04T07:10:02Z" level=info msg="package(s) specified are - percona,postgresql,strimzi-kafka-operator,node-problem-detector,automationbroker,kiecloud-operator,elasticsearch-operator,node-network-operator,microcks,prometheus,jaeger,cluster-logging,planetscale,cockroachdb,metering,descheduler,oneagent,templateservicebroker,federation,etcd,camel-k" port=50051 type=appregistry
  | time="2019-04-04T07:10:02Z" level=info msg="can't proceed, bailing out" port=50051 type=appregistry
  | time="2019-04-04T07:10:02Z" level=error msg="the following error(s) occurred while parsing input - invalid source, secret specified is malformed - upstream-community-operators" port=50051 type=appregistry
  | time="2019-04-04T07:10:02Z" level=fatal msg="error loading manifest from remote registry - invalid source, secret specified is malformed - upstream-community-operators" port=50051 type=appregistry

Pod deployed

apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/scc: restricted
  creationTimestamp: '2019-04-04T06:53:28Z'
  generateName: community-operators-664fc79df5-
  labels:
    marketplace.catalogSourceConfig: community-operators
    pod-template-hash: '2209735891'
  name: community-operators-664fc79df5-j8n92
  namespace: marketplace
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: community-operators-664fc79df5
      uid: 5a36e2c0-56a6-11e9-b581-080027f60d62
  resourceVersion: '31663'
  selfLink: /api/v1/namespaces/marketplace/pods/community-operators-664fc79df5-j8n92
  uid: 5a3c5ef5-56a6-11e9-b581-080027f60d62
spec:
  containers:
    - command:
        - appregistry-server
        - >-
          --registry=https://quay.io/cnr|community-operators
          --registry=https://quay.io/cnr|upstream-community-operators
        - '-o'
        - >-
          percona,postgresql,strimzi-kafka-operator,node-problem-detector,automationbroker,kiecloud-operator,elasticsearch-operator,node-network-operator,microcks,prometheus,jaeger,cluster-logging,planetscale,cockroachdb,metering,descheduler,oneagent,templateservicebroker,federation,etcd,camel-k
      image: quay.io/openshift/origin-operator-registry
      imagePullPolicy: Always
      livenessProbe:
        exec:
          command:
            - grpc_health_probe
            - '-addr=localhost:50051'
        failureThreshold: 30
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: community-operators
      ports:
        - containerPort: 50051
          name: grpc
          protocol: TCP
      readinessProbe:
        exec:
          command:
            - grpc_health_probe
            - '-addr=localhost:50051'
        failureThreshold: 30
        initialDelaySeconds: 5
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources: {}
      securityContext:
        capabilities:
          drop:
            - KILL
            - MKNOD
            - SETGID
            - SETUID
        runAsUser: 1000200000
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: default-token-pgrsw
          readOnly: true
  dnsPolicy: ClusterFirst
  imagePullSecrets:
    - name: default-dockercfg-kkjkn
  nodeName: localhost
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 1000200000
    seLinuxOptions:
      level: 's0:c14,c9'
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
    - name: default-token-pgrsw
      secret:
        defaultMode: 420
        secretName: default-token-pgrsw
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2019-04-04T06:53:29Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2019-04-04T06:53:29Z'
      message: 'containers with unready status: [community-operators]'
      reason: ContainersNotReady
      status: 'False'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: null
      message: 'containers with unready status: [community-operators]'
      reason: ContainersNotReady
      status: 'False'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2019-04-04T06:53:29Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        docker://26d42781129f924cf8586848d3dff3665056ef16f0198c3df6c53f9db7f36b7a
      image: 'quay.io/openshift/origin-operator-registry:latest'
      imageID: >-
        docker-pullable://quay.io/openshift/origin-operator-registry@sha256:aa424fce55ad5a28b19fcc9399cba6d62bc04a6d9a4730dfd150e9b36d4262f6
      lastState:
        terminated:
          containerID: >-
            docker://26d42781129f924cf8586848d3dff3665056ef16f0198c3df6c53f9db7f36b7a
          exitCode: 1
          finishedAt: '2019-04-04T07:15:18Z'
          message: >
            time="2019-04-04T07:15:18Z" level=fatal msg="error loading manifest
            from remote registry - invalid source, secret specified is malformed
            - upstream-community-operators" port=50051 type=appregistry
          reason: Error
          startedAt: '2019-04-04T07:15:18Z'
      name: community-operators
      ready: false
      restartCount: 9
      state:
        waiting:
          message: >-
            Back-off 5m0s restarting failed container=community-operators
            pod=community-operators-664fc79df5-j8n92_marketplace(5a3c5ef5-56a6-11e9-b581-080027f60d62)
          reason: CrashLoopBackOff
  hostIP: 10.0.3.15
  phase: Running
  podIP: 172.17.0.13
  qosClass: BestEffort
  startTime: '2019-04-04T06:53:29Z'

Answer 1 · 2019-04-04T14:11:36.000Z

Please see #154 (comment)

Answer 2 · 2019-04-04T14:59:00.000Z

I did a new test and followed your suggestion and that works

time="2019-04-04T14:50:35Z" level=info msg="Using in-cluster kube client config" port=50051 type=appregistry
time="2019-04-04T14:50:35Z" level=info msg="operator source(s) specified are - [https://quay.io/cnr|upstream-community-operators ]" port=50051 type=appregistry
time="2019-04-04T14:50:35Z" level=info msg="package(s) specified are - federation,etcd,couchbase-enterprise,mongodb-enterprise,planetscale,redis-enterprise,strimzi-kafka-operator,microcks,percona,oneagent,cockroachdb,jaeger,aws-service,prometheus,postgresql,vault" port=50051 type=appregistry
time="2019-04-04T14:50:35Z" level=info msg="input has been sanitized" port=50051 type=appregistry
time="2019-04-04T14:50:35Z" level=info msg="sources: [https://quay.io/cnr/upstream-community-operators ]" port=50051 type=appregistry
time="2019-04-04T14:50:35Z" level=info msg="packages: [federation etcd couchbase-enterprise mongodb-enterprise planetscale redis-enterprise strimzi-kafka-operator microcks percona oneagent cockroachdb jaeger aws-service prometheus postgresql vault]" port=50051 type=appregistry
time="2019-04-04T14:50:36Z" level=info msg="resolved the following packages: [upstream-community-operators/postgresql:0.0.2 upstream-community-operators/federation:0.0.2 upstream-community-operators/vault:0.0.1 upstream-community-operators/jaeger:0.0.2 upstream-community-operators/aws-service:0.0.3 upstream-community-operators/redis-enterprise:0.0.2 upstream-community-operators/strimzi-kafka-operator:0.0.3 upstream-community-operators/etcd:0.0.5 upstream-community-operators/prometheus:0.0.1 upstream-community-operators/couchbase-enterprise:0.0.5 upstream-community-operators/cockroachdb:0.0.1 upstream-community-operators/oneagent:0.0.1 upstream-community-operators/microcks:0.0.1 upstream-community-operators/percona:0.0.4 upstream-community-operators/mongodb-enterprise:0.0.2 upstream-community-operators/planetscale:0.0.2]" port=50051 type=appregistry
time="2019-04-04T14:50:36Z" level=info msg="downloading repository: upstream-community-operators/postgresql:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:39Z" level=info msg="downloading repository: upstream-community-operators/federation:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:40Z" level=info msg="downloading repository: upstream-community-operators/vault:0.0.1 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:42Z" level=info msg="downloading repository: upstream-community-operators/jaeger:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:45Z" level=info msg="downloading repository: upstream-community-operators/aws-service:0.0.3 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:46Z" level=info msg="downloading repository: upstream-community-operators/redis-enterprise:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:48Z" level=info msg="downloading repository: upstream-community-operators/strimzi-kafka-operator:0.0.3 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:50Z" level=info msg="downloading repository: upstream-community-operators/etcd:0.0.5 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:53Z" level=info msg="downloading repository: upstream-community-operators/prometheus:0.0.1 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:54Z" level=info msg="downloading repository: upstream-community-operators/couchbase-enterprise:0.0.5 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:56Z" level=info msg="downloading repository: upstream-community-operators/cockroachdb:0.0.1 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:50:59Z" level=info msg="downloading repository: upstream-community-operators/oneagent:0.0.1 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:51:01Z" level=info msg="downloading repository: upstream-community-operators/microcks:0.0.1 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:51:03Z" level=info msg="downloading repository: upstream-community-operators/percona:0.0.4 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:51:05Z" level=info msg="downloading repository: upstream-community-operators/mongodb-enterprise:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:51:07Z" level=info msg="downloading repository: upstream-community-operators/planetscale:0.0.2 from https://quay.io/cnr " port=50051 type=appregistry
time="2019-04-04T14:51:09Z" level=info msg="download complete - 16 repositories have been downloaded" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="all manifest(s) have been merged into one" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="loading into sqlite database" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="using configmap loader to build sqlite database" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="loading CRDs" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="loading Bundles" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="loading Packages" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="extracting provided API information" port=50051 type=appregistry
time="2019-04-04T14:51:10Z" level=info msg="serving registry" port=50051 type=appregistry