Provide for reusing PV in stateful stream apps.
corneil opened this issue · 3 comments
Users need to specify the volumeName.
Users need to be able to derive mapped path names from stream/app names.
Users need to configure a dataSource for the PVC. They can then reference an existing PVC or VolumeSnapshot.
But how would this work with dynamic provisioning, especially when you have a versioned deployment.
When we deploy Stateful KStream Apps (Using Kafka Streams Binder) with SCDF as StatefulSets, the name of the StatefulSet includes the name of the stream, the name of the app and a version number. This same name is used to create the name of the underlying PVC.
When a stream is updated with new versions, a new PV is created and a new subpath will be created, the PVC will also have a new name with a new version number. I know we can manually change the name back, but that means whenever there is a new deployment we need to do a manually change, that will be too much work when we have all several hundred apps migrated to k8s.
We are considering an option where updates patches existing deployment/statefulset instead of creating a new version.
Currently the PVC name is combination of VolumeTemplate name + Statefulset, volumetemplatename is externalized, can the add prefix be externalized, so if user specifies one use it else use statefulset?
The idea primarily is to ensure when we use stream apps that do read data from kafka topics and store it to local to perform stateful kafka operations, it doesnt have to do a full download of data from kafka topic in event of new deploys.