vitessio/vitess-operator

Endless loop at “init-replica-master” job

MasayaAoyama opened this issue · 0 comments

MasayaAoyama commented

I use following VitessCluster CRD, I specify "spec.keyspaces[].spec.shards[].spec.defaults.replicas" to 5.

## Sample VitessCluster all-in-one resource
apiVersion: vitess.io/v1alpha2
kind: VitessCluster
metadata:
  name: aio
  labels:
    app: vitess
spec:
  lockserver:
    metadata:
      name: global
    spec:
      type: etcd2
      etcd2:
        address: etcd-global-client:2379
        pathPrefix: /vitess/global
  cells:
  - metadata:
      name: zone1
    spec:
      lockserver:
        metadata:
          name: zone1
        spec:
          type: etcd2
          etcd2:
            address: etcd-zone1-client:2379
            pathPrefix: /vitess/zone1
      defaults:
        replicas: 1
        image: vitess/vttablet:helm-1.0.6
  keyspaces:
  - metadata:
      name: sharded-dbname
    spec:
      shards:
      - metadata:
          name: "x-80"
        spec:
          keyRange: { to: "80" }
          defaults:
            replicas: 2
            containers:
              mysql:
                image: percona:5.7.23
              vttablet:
                image: vitess/vttablet:helm-1.0.6
          tablets:
          - metadata:
              name: zone1
            spec:
              cellID: zone1
              tabletID: 102
              type: replica
      - metadata:
          name: "80-x"
        spec:
          keyRange: { from: "80" }
          defaults:
            replicas: 5
            containers:
              mysql:
                image: percona:5.7.23
              vttablet:
                image: vitess/vttablet:helm-1.0.6
          tablets:
          - metadata:
              name: zone1
            spec:
              cellID: zone1
              tabletID: 103
              type: replica

But "aio-zone1-sharded-dbname-80-x-init-replica-master" job is not completed, job's logs said following.

++ echo 'zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ awk '{print $1}'
+ tabletCount=5
+ '[' 5 == 2 ']'
+ ((  115 > 600  ))
+ sleep 5
+ '[' ']'
++ vtctlclient -server aio-zone1-vtctld.default:15999 ListAllTablets zone1
+ cellTablets='zone1-0638444000 sharded-dbname -80 master aio-zone1-sharded-dbname-x-80-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-x-80-replica-0.aio-tab:3306 []
zone1-0638444001 sharded-dbname -80 replica aio-zone1-sharded-dbname-x-80-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-x-80-replica-1.aio-tab:3306 []
zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ echo 'zone1-0638444000 sharded-dbname -80 master aio-zone1-sharded-dbname-x-80-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-x-80-replica-0.aio-tab:3306 []
zone1-0638444001 sharded-dbname -80 replica aio-zone1-sharded-dbname-x-80-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-x-80-replica-1.aio-tab:3306 []
zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ grep -w aio-zone1-sharded-dbname-80-x
+ shardTablets='zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ echo 'zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ awk '$4 == "master" {print $1}'
+ masterTablet=
+ '[' ']'
++ jq .master_alias.uid
++ vtctlclient -server aio-zone1-vtctld.default:15999 GetShard sharded-dbname/80-
+ master_alias=null
+ '[' null '!=' null -a null '!=' '' ']'
++ wc
++ echo 'zone1-1676054800 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-0.aio-tab:3306 []
zone1-1676054801 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-1.aio-tab:3306 []
zone1-1676054802 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-2.aio-tab:3306 []
zone1-1676054803 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-3.aio-tab:3306 []
zone1-1676054804 sharded-dbname 80- replica aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:15002 aio-zone1-sharded-dbname-80-x-replica-4.aio-tab:3306 []'
++ awk '{print $1}'
+ tabletCount=5
+ '[' 5 == 2 ']'
+ ((  120 > 600  ))
+ sleep 5