Kubespray periodic backup etcd data - disaster recovery cluster from etcd backups

[haminhcong]

• Which Kubernetes cluster data stored in ETCD
• Monitor etcd data backup process and alert when failed to backup K8S data
• When recovery, do we need change or keep master IP and Name ?
• Persistent Volume Backup & Restore ? https://github.com/vmware-tanzu/velero

Data which need to save and backup periodic from K8S Cluster Provisioned By Kubespray

• Inventory File
• Node Password Information
• Etcd Node Data:
  • Etcd backup
  • Etcd certificates

Backup And Monitoring Tools

• https://github.com/gardener/etcd-backup-restore/blob/master/doc/usage/metrics.md

Possible scenario

• Recover etcd cluster from backup data
• Recover first master node
• Join remaining master nodes
• Join remaining worker nodes

Need to search in kubespray slack about this problem and this playbook kubernetes-sigs/kubespray@ac2135e#diff-3bda861e9e89b5053f64df966fe350fcbbe32a3c5f1b977fdb8f004e28c7d47b

References

• https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
• kubernetes-sigs/kubespray#4146
• kubernetes-sigs/kubespray#1122
• https://documentation.suse.com/suse-caasp/4.2/html/caasp-admin/_cluster_disaster_recovery.html