Notebook volumes disappears after the server with Kubeflow restart (same with K8s cluster scaling/eviction)
Closed this issue · 2 comments
Barteus commented
Reproduce:
- Install Kubeflow according to the "Quick guide"
- Create a notebook in UI and do some work on it (create a few files, run some notebooks)
- Restart the machine with Kubeflow
- Go back to the Jupyter notebook - it does not have the work we did in it.
Expected:
The volume that is attached preserves the
Workaround:
Use Git frequently.
Environment:
Microk8s: 1.22
Kubeflow: 1.6
i-chvets commented
Verified with Kubeflow 1.6 K8S 1.24.
After restart of VM with MicroK8S cluster all service came up and connection to notebook server could be made.
Details of deployment:
| microk8s status | ||||
|---|---|---|---|---|
| dns | ||||
| ha-cluster | ||||
| hostpath-storage | ||||
| ingress | ||||
| metallb | ||||
| storage | ||||
| juju status | ||||
| App | Version | Charm | Channel | Rev |
| admission-webhook | res:oci-image@129fe92 | admission-webhook | 1.6/stable | 60 |
| argo-controller | res:oci-image@669ebd5 | argo-controller | 3.3/stable | 99 |
| argo-server | res:oci-image@576d038 | argo-server | 3.3/stable | 45 |
| dex-auth | dex-auth | 2.31/stable | 129 | |
| istio-ingressgateway | istio-gateway | 1.11/stable | 114 | |
| istio-pilot | istio-pilot | 1.11/stable | 131 | |
| jupyter-controller | res:oci-image@e05857e | jupyter-controller | 1.6/stable | 163 |
| jupyter-ui | res:oci-image@d55c600 | jupyter-ui | 1.6/stable | 124 |
| katib-controller | res:oci-image@03d47fb | katib-controller | 0.14/stable | 92 |
| katib-db | mariadb/server:10.3 | charmed-osm-mariadb-k8s | latest/stable | 35 |
| katib-db-manager | res:oci-image@16b33a5 | katib-db-manager | 0.14/stable | 66 |
| katib-ui | res:oci-image@c7dc04a | katib-ui | 0.14/stable | 90 |
| kfp-api | res:oci-image@bf747d5 | kfp-api | 2.0/stable | 144 |
| kfp-db | mariadb/server:10.3 | charmed-osm-mariadb-k8s | latest/stable | 35 |
| kfp-persistence | res:oci-image@abcf971 | kfp-persistence | 2.0/stable | 141 |
| kfp-profile-controller | res:oci-image@b4de878 | kfp-profile-controller | 2.0/stable | 125 |
| kfp-schedwf | res:oci-image@9c9f710 | kfp-schedwf | 2.0/stable | 155 |
| kfp-ui | res:oci-image@47864af | kfp-ui | 2.0/stable | 144 |
| kfp-viewer | res:oci-image@94754c0 | kfp-viewer | 2.0/stable | 152 |
| kfp-viz | res:oci-image@23ab9b9 | kfp-viz | 2.0/stable | 134 |
| kubeflow-dashboard | res:oci-image@6fe6eec | kubeflow-dashboard | 1.6/stable | 183 |
| kubeflow-profiles | res:profile-image@cfd6935 | kubeflow-profiles | 1.6/stable | 94 |
| kubeflow-roles | kubeflow-roles | 1.6/stable | 49 | |
| kubeflow-volumes | res:oci-image@fdb4a5d | kubeflow-volumes | 1.6/stable | 84 |
| metacontroller-operator | metacontroller-operator | 2.0/stable | 48 | |
| minio | res:oci-image@1755999 | minio | ckf-1.6/stable | 99 |
| mlflow-db | mariadb/server:10.3 | charmed-osm-mariadb-k8s | stable | 35 |
| mlflow-server | res:oci-image@bba33cd | mlflow-server | stable | 77 |
| oidc-gatekeeper | res:oci-image@32de216 | oidc-gatekeeper | ckf-1.6/stable | 76 |
| seldon-controller-manager | res:oci-image@eb811b6 | seldon-core | 1.14/stable | 92 |
| tensorboard-controller | res:oci-image@51058f7 | tensorboard-controller | 1.6/stable | 69 |
| tensorboards-web-app | res:oci-image@eef68a5 | tensorboards-web-app | 1.6/stable | 71 |
| training-operator | training-operator | 1.5/stable | 65 |
i-chvets commented
If problem occurs again. New issue will be opened. Closing.