operator tries to shrink volume after storage size update if value is in Mi
Opened this issue · 5 comments
mahdiG commented
Bug Report
What version of Kubernetes are you using?
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.6
What version of TiDB Operator are you using?
TiDB Operator Version: version.Info{GitVersion:"v1.5.3", GitCommit:"2c9e4dad0abaa4400afdef9ceff3084e71510ecb", GitTreeState:"clean", BuildDate:"2024-04-18T03:46:15Z", GoVersion:"go1.21.6", Compiler:"gc", Platform:"linux/amd64"}
What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?
cephfs.csi.ceph.com
What's the status of the TiDB cluster pods?
Running
What did you do?
I tried to update storage size in TidbCluster for PD and TiKV to 1740Mi and 15675Mi. The operator updated the volumes correctly but then in operator logs, I get these errors:
1 phase.go:74] volume rvgnrvpp/tikv-pnhrmptw-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
1 phase.go:74] volume rvgnrvpp/pd-pnhrmptw-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I set the disk size in Mi but after the volumes are updated, the operator seems to round up and use Gi and then complain about not being able to shrink the disk size from Gi value to Mi value that I've set.
What did you expect to see?
I expected the operator NOT to try to shrink volumes after it updated the volumes.
What did you see instead?
I saw the operator incorrectly assuming the storage size is 16Gi instead of 15675Mi and try to shrink the volumes.
csuzhangxc commented
Can you show the size of your PVCs (both TiKV and PD)?
mahdiG commented
The problem happens on create too, not just on update. I created a new TidbCluster and here's all the manifests and logs:
I created TidbCluster CR with the same storage sizes as before (15675Mi and 1740Mi). Here's the CR copied from Lens after creation:
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
creationTimestamp: '2024-05-20T06:26:04Z'
generation: 16
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
name: wtiagtmv
namespace: rvgnrvpp
resourceVersion: '572028224'
uid: 77ce4dd6-fb5d-4814-9ca7-96393bec6d94
selfLink: /apis/pingcap.com/v1alpha1/namespaces/rvgnrvpp/tidbclusters/wtiagtmv
status:
clusterID: '7370964382394004606'
conditions:
- lastTransitionTime: '2024-05-20T06:28:23Z'
lastUpdateTime: '2024-05-20T06:28:23Z'
message: TiDB cluster is fully up and running
reason: Ready
status: 'True'
type: Ready
pd:
image: repo.lexoya.com/pingcap/pd:v7.5.1
leader:
clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
health: true
id: '10249660128629638972'
lastTransitionTime: '2024-05-20T06:26:27Z'
name: wtiagtmv-pd-0
members:
wtiagtmv-pd-0:
clientURL: http://wtiagtmv-pd-0.wtiagtmv-pd-peer.rvgnrvpp.svc:2379
health: true
id: '10249660128629638972'
lastTransitionTime: '2024-05-20T06:26:27Z'
name: wtiagtmv-pd-0
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-pd-6597dfcbd4
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-pd-6597dfcbd4
updatedReplicas: 1
synced: true
volumes:
pd:
boundCount: 1
currentCapacity: 2Gi
currentCount: 1
currentStorageClass: csi-cephfs-sc-nvme
modifiedCapacity: 1740Mi
modifiedCount: 0
modifiedStorageClass: csi-cephfs-sc-nvme
name: pd
resizedCapacity: 1740Mi
resizedCount: 0
pump: {}
ticdc: {}
tidb:
image: repo.lexoya.com/pingcap/tidb:v7.5.1
members:
wtiagtmv-tidb-0:
health: true
lastTransitionTime: '2024-05-20T06:28:23Z'
name: wtiagtmv-tidb-0
node: 192.168.110.158
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-tidb-55f66f78b4
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-tidb-55f66f78b4
updatedReplicas: 1
tiflash: {}
tikv:
bootStrapped: true
conditions:
- lastTransitionTime: '2024-05-20T06:27:13Z'
message: Leader can be scheduled to all nodes
reason: NoLeaderEviction
status: 'False'
type: LeaderEvicting
image: repo.lexoya.com/pingcap/tikv:v7.5.1
phase: Normal
statefulSet:
collisionCount: 0
currentReplicas: 1
currentRevision: wtiagtmv-tikv-5464f9546f
observedGeneration: 1
readyReplicas: 1
replicas: 1
updateRevision: wtiagtmv-tikv-5464f9546f
updatedReplicas: 1
stores:
'1':
id: '1'
ip: wtiagtmv-tikv-0.wtiagtmv-tikv-peer.rvgnrvpp.svc
lastTransitionTime: '2024-05-20T06:27:12Z'
leaderCount: 60
podName: wtiagtmv-tikv-0
state: Up
synced: true
volumes:
tikv:
boundCount: 1
currentCapacity: 16Gi
currentCount: 1
currentStorageClass: csi-cephfs-sc-nvme
modifiedCapacity: 15675Mi
modifiedCount: 0
modifiedStorageClass: csi-cephfs-sc-nvme
name: tikv
resizedCapacity: 15675Mi
resizedCount: 0
tiproxy: {}
spec:
configUpdateStrategy: RollingUpdate
discovery:
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
limits:
cpu: 50m
memory: 100Mi
requests:
cpu: 1m
memory: 20Mi
enableDynamicConfiguration: true
enablePVReclaim: false
helper:
image: repo.lexoya.com/library/alpine:3.16.0
imagePullPolicy: IfNotPresent
labels:
lexoya/component: '25'
lexoya/name: wtiagtmv
lexoya/version: '69'
pd:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- pd
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/pd
config: |
[dashboard]
internal-proxy = true
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[replication]
max-replicas = 1
limits:
cpu: '1.318'
memory: 223M
maxFailoverCount: 3
mountClusterClientSecret: true
replicas: 1
requests:
cpu: 180m
memory: 120Mi
storage: 1740Mi
startTimeout: 30
pvReclaimPolicy: Delete
startScriptVersion: v2
suspendAction: {}
tidb:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- tidb
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/tidb
config: |
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[performance]
tcp-keep-alive = true
limits:
cpu: 467m
memory: 1276M
maxFailoverCount: 3
replicas: 1
requests:
cpu: 100m
memory: 200Mi
service:
type: ClusterIP
tikv:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values:
- tikv
- key: lexoya/name
operator: In
values:
- wtiagtmv
topologyKey: kubernetes.io/hostname
weight: 1
baseImage: repo.lexoya.com/pingcap/tikv
config: |
log-level = "info"
[log]
[log.file]
max-backups = 1
max-days = 1
max-size = 100
[raftdb]
max-total-wal-size = "1GB"
[rocksdb]
max-total-wal-size = "1GB"
[storage]
reserve-space = "100MB"
limits:
cpu: 467m
memory: 4511M
maxFailoverCount: 3
mountClusterClientSecret: true
replicas: 1
requests:
cpu: 100m
memory: 1536Mi
storage: 15675Mi
scalePolicy:
scaleInParallelism: 1
scaleOutParallelism: 1
timezone: UTC
tlsCluster: {}
version: v7.5.1
TiKV pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tikv-wtiagtmv-tikv-0
namespace: rvgnrvpp
uid: 9d15043b-0636-45bb-9a1b-b1af1ab02948
resourceVersion: '572027744'
creationTimestamp: '2024-05-20T06:26:28Z'
labels:
app.kubernetes.io/component: tikv
app.kubernetes.io/instance: wtiagtmv
app.kubernetes.io/managed-by: tidb-operator
app.kubernetes.io/name: tidb-cluster
tidb.pingcap.com/cluster-id: '7370964382394004606'
tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
tidb.pingcap.com/store-id: '1'
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
tidb.pingcap.com/pod-name: wtiagtmv-tikv-0
volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
finalizers:
- kubernetes.io/pvc-protection
selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/tikv-wtiagtmv-tikv-0
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 16Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15675Mi
volumeName: pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948
storageClassName: csi-cephfs-sc-nvme
volumeMode: Filesystem
PD pvc:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pd-wtiagtmv-pd-0
namespace: rvgnrvpp
uid: ea461780-f79f-47b8-947b-f2701ce5d192
resourceVersion: '572027731'
creationTimestamp: '2024-05-20T06:26:06Z'
labels:
app.kubernetes.io/component: pd
app.kubernetes.io/instance: wtiagtmv
app.kubernetes.io/managed-by: tidb-operator
app.kubernetes.io/name: tidb-cluster
tidb.pingcap.com/cluster-id: '7370964382394004606'
tidb.pingcap.com/member-id: '10249660128629638972'
tidb.pingcap.com/pod-name: wtiagtmv-pd-0
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
tidb.pingcap.com/pod-name: wtiagtmv-pd-0
volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
finalizers:
- kubernetes.io/pvc-protection
selfLink: /api/v1/namespaces/rvgnrvpp/persistentvolumeclaims/pd-wtiagtmv-pd-0
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 2Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1740Mi
volumeName: pvc-ea461780-f79f-47b8-947b-f2701ce5d192
storageClassName: csi-cephfs-sc-nvme
volumeMode: Filesystem
operator logs:
E0520 06:26:05.033840 1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: TidbInitManager.Sync: failed to get tidbcluster wtiagtmv for TidbInitializer rvgnrvpp/wtiagtmv, error: tidbcluster.pingcap.com "wtiagtmv" not found, requeuing
I0520 06:26:05.318603 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-tc-client-tls for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.318695 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.319347 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-dashboard-exposed in wtiagtmv successful
I0520 06:26:05.353237 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.357424 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbDashboard", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"b39fbc16-cdb9-4a15-bf3a-c7a2e02f6d03", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026967", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb-dashboard in wtiagtmv successful
I0520 06:26:05.365138 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-ng-monitoring in wtiagtmv successful
I0520 06:26:05.393694 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-ng-monitoring-3961393 for controller TidbNGMonitoring/wtiagtmv successfully
I0520 06:26:05.417535 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbNGMonitoring", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"815b5ddc-a8c3-44a1-bbf7-2cd0420030c3", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026966", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-ng-monitoring in wtiagtmv successful
I0520 06:26:05.512125 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.544550 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Deployment/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.596824 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Service/wtiagtmv-discovery for controller TidbCluster/wtiagtmv successfully
I0520 06:26:05.859607 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd in wtiagtmv successful
I0520 06:26:06.061752 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
E0520 06:26:06.088485 1 tidb_initializer_controller.go:106] TiDBInitializer: rvgnrvpp/wtiagtmv, sync failed, err: updateStatus: failed to get job wtiagtmv-tidb-initializer for TidbInitializer rvgnrvpp/wtiagtmv, error: job.batch "wtiagtmv-tidb-initializer" not found, requeuing
I0520 06:26:06.088665 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbInitializer", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"1fc1a34b-60d5-4c97-823c-6d2358e6378b", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026963", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Job/wtiagtmv-tidb-initializer for controller TidbInitializer/wtiagtmv successfully
I0520 06:26:06.242077 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-pd-peer in wtiagtmv successful
I0520 06:26:06.273267 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-pd-6334666 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:06.301665 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026962", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-pd in wtiagtmv successful
I0520 06:26:06.349515 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:06.349721 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for PD cluster running, requeuing
E0520 06:26:06.813751 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
I0520 06:26:06.943683 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:06.943885 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.348161 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp: lookup wtiagtmv-pd.rvgnrvpp on 10.43.0.10:53: no such host, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.387432 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:07.723528 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:07.761553 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:08.125232 1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
{"level":"warn","ts":"2024-05-20T06:26:10.220Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:10.220875 1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:10.220905 1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:10.230130 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/tidbmonitor-wtiagtmv-tls-assets for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.265579 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-prometheus in wtiagtmv successful
I0520 06:26:10.300232 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-monitor-reloader in wtiagtmv successful
I0520 06:26:10.373574 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-grafana in wtiagtmv successful
I0520 06:26:10.400498 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.414949 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-monitor-grafana for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.431792 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Secret/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.450916 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ServiceAccount/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.463887 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create Role/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
I0520 06:26:10.512205 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create RoleBinding/wtiagtmv-monitor for controller TidbMonitor/wtiagtmv successfully
W0520 06:26:10.544937 1 warnings.go:70] spec.template.spec.initContainers[0].env[5].name: duplicate name "TZ"
I0520 06:26:10.545723 1 tidb_monitor_controller.go:99] TidbMonitor: rvgnrvpp/wtiagtmv, still need sync: TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running, requeuing
I0520 06:26:10.545752 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-monitor in wtiagtmv successful
I0520 06:26:10.545822 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbMonitor", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"48840cfc-d376-4e3e-8390-13e0c6c7089f", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572026964", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Sync TidbMonitor[rvgnrvpp/wtiagtmv] Statefulset failed, err:TidbMonitor: [rvgnrvpp/wtiagtmv], waiting for tidbmonitor running
E0520 06:26:11.662535 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:11.692055 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:12.243252 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused, service rvgnrvpp/wtiagtmv-pd has no endpoints
E0520 06:26:12.282986 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
E0520 06:26:15.031799 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
I0520 06:26:15.374715 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:26:15.374880 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
I0520 06:26:15.506073 1 tidb_dashboard_control.go:134] TidbDashboard: [rvgnrvpp/wtiagtmv], update status successfully
E0520 06:26:15.689451 1 pd_member_manager.go:196] failed to sync TidbCluster: [rvgnrvpp/wtiagtmv]'s status, error: Get "http://wtiagtmv-pd.rvgnrvpp:2379/pd/api/v1/health": dial tcp 10.43.137.11:2379: connect: connection refused
E0520 06:26:15.729608 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed TidbCluster: rvgnrvpp/wtiagtmv .Status.PD.Synced = false, can't failover, requeuing
{"level":"warn","ts":"2024-05-20T06:26:16.546Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-23807697-7bf5-4485-93c1-93cb67f81e10/wtiagtmv-pd.rvgnrvpp:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.43.137.11:2379: connect: connection refused\""}
E0520 06:26:16.547071 1 monitor_manager.go:786] context deadline exceeded
E0520 06:26:16.547108 1 monitor_manager.go:107] Fail to sync TiDB Dashboard metrics config for TiDB cluster [rvgnrvpp/wtiagtmv], error: context deadline exceeded
I0520 06:26:16.859702 1 pv_control.go:192] PV: [pvc-e9d36715-8a9e-41d5-aec7-5cd142c33e46] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:26:16.893560 1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:20.736268 1 tidb_ng_monitoring_control.go:153] TidbNGMonitoring: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:26:27.956642 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.002556 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tikv-peer in wtiagtmv successful
I0520 06:26:28.032478 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tikv-6664313 for controller TidbCluster/wtiagtmv successfully
I0520 06:26:28.060629 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027294", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tikv in wtiagtmv successful
I0520 06:26:28.121069 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.121242 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:28.624818 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:28.665325 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:28.696456 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:28.763600 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:28.763730 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:29.094536 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:29.135019 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:29.187116 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:29.188874 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.048099 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.112530 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.149937 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.213674 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:34.213802 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:34.648952 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
I0520 06:26:34.707919 1 tikv_member_manager.go:851] TiKV of Cluster rvgnrvpp/wtiagtmv not bootstrapped yet
I0520 06:26:34.745607 1 tikv_member_manager.go:965] TiKV of Cluster rvgnrvpp/wtiagtmv is not bootstrapped yet, no need to set store labels
I0520 06:26:34.747280 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.170709 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.212448 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.310081 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
I0520 06:26:42.310189 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:26:42.596586 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:26:42.641823 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:26:42.686692 1 tidb_cluster_controller.go:141] TidbCluster: rvgnrvpp/wtiagtmv, still need sync: TidbCluster: [rvgnrvpp/wtiagtmv], waiting for TiKV cluster running, requeuing
W0520 06:27:12.215575 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:12.263871 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:12.325857 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb-peer in wtiagtmv successful
I0520 06:27:12.375332 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create Service wtiagtmv-tidb in wtiagtmv successful
I0520 06:27:12.401228 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'Successfully Create' create ConfigMap/wtiagtmv-tidb-6436643 for controller TidbCluster/wtiagtmv successfully
I0520 06:27:12.433168 1 event.go:282] Event(v1.ObjectReference{Kind:"TidbCluster", Namespace:"rvgnrvpp", Name:"wtiagtmv", UID:"77ce4dd6-fb5d-4814-9ca7-96393bec6d94", APIVersion:"pingcap.com/v1alpha1", ResourceVersion:"572027531", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' create StatefulSet wtiagtmv-tidb in wtiagtmv successful
I0520 06:27:12.558263 1 pv_control.go:192] PV: [pvc-ea461780-f79f-47b8-947b-f2701ce5d192] updated successfully, : rvgnrvpp/wtiagtmv
I0520 06:27:12.926823 1 pv_control.go:192] PV: [pvc-9d15043b-0636-45bb-9a1b-b1af1ab02948] updated successfully, : rvgnrvpp/wtiagtmv
W0520 06:27:13.139695 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.140055 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.232600 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.232799 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed [modify volumes for rvgnrvpp/wtiagtmv:tidb failed: component phase is not Normal, modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv], requeuing
W0520 06:27:13.529859 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.574867 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:13.829302 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:13.829678 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:13.983249 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
E0520 06:27:13.983419 1 tidb_cluster_controller.go:143] TidbCluster: rvgnrvpp/wtiagtmv, sync failed modify volumes for rvgnrvpp/wtiagtmv:tikv failed: try to stop evicting leader for tidbcluster rvgnrvpp/wtiagtmv, requeuing
W0520 06:27:14.283262 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.326888 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:14.427504 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:14.427768 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:14.555305 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:15.029004 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.076644 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.281218 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.281579 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:15.605736 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:15.677335 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:16.352707 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:16.352959 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.231406 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.281047 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:42.442824 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.443135 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:42.528256 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:42.818005 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:42.921079 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.010650 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.010925 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:27:43.084599 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:27:43.300293 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.335028 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:27:43.850035 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:27:43.850247 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:02.520831 1 tidb_monitor_control.go:120] TidbMonitor: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:08.451478 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.501857 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:08.611842 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:08.612097 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:08.707193 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:09.222522 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.261375 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:09.349822 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:09.350090 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.245023 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.291253 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:12.398556 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.398755 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:12.544318 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:12.893828 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:12.953947 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:13.058740 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:13.058939 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.202208 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.242786 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.352464 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.352565 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:23.421251 1 tidbcluster_control.go:71] TidbCluster: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:28:23.702868 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.743086 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:23.824304 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:23.824630 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.218587 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.266987 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:28:42.374751 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:28:42.374974 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
I0520 06:28:48.912223 1 tidb_init_manager.go:142] TidbInitializer: [rvgnrvpp/wtiagtmv] updated successfully
W0520 06:29:12.221467 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.273803 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:12.370491 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:12.370823 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.178022 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.220272 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
W0520 06:29:42.308498 1 phase.go:74] volume rvgnrvpp/pd-wtiagtmv-pd-0 modification is not allowed: can't shrunk size from 2Gi to 1740Mi
W0520 06:29:42.308846 1 phase.go:74] volume rvgnrvpp/tikv-wtiagtmv-tikv-0 modification is not allowed: can't shrunk size from 16Gi to 15675Mi
You can ignore logs for other CRs. The TidbInitializer only creates an "app" database and does nothing else.
As you can see, operator creates the cluster with correct pvc sizes but then tries to shrink from a higher size to the original correct size.
csuzhangxc commented
from PVC
status:
phase: Bound
accessModes:
- ReadWriteOnce
capacity:
storage: 16Gi
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 15675MiIt seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
mahdiG commented
from PVC
status: phase: Bound accessModes: - ReadWriteOnce capacity: storage: 16Gi spec: accessModes: - ReadWriteOnce resources: requests: storage: 15675MiIt seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status
csuzhangxc commented
from PVC
status: phase: Bound accessModes: - ReadWriteOnce capacity: storage: 16Gi spec: accessModes: - ReadWriteOnce resources: requests: storage: 15675MiIt seems the CSI driver created a 16Gi volume instead of a 15675Mi one.
Interesting. Maybe if operator looks at pvc spec.resources.requests.storage which is 15675Mi instead of pvc status.capacity.storage which is 16Gi, the problem is fixed. I'm not an expert but I think spec is a more solid thing to look at than status
When reszing a PVC, operator need to check the status to ensure whether the expanding operation is finished.