TFX for small, single-laptop workflows
SamuelMarks opened this issue · 4 comments
Describe the feature and the current behavior/state.
Apache Beam, Apache Airflow, Kubeflow are all large components; reliable; and useful at scale.
But what about for the pedestrian target of your local development laptop? - Maybe even air-gapped?
Will this change the current API? How?
It would involve supporting a new alternative to Apache Beam, Apache Airflow, and Kubeflow. A solution which is small database based—e.g., SQLite, etcd—and that has both small memory and disk footprint.
Who will benefit with this feature?
It would help in quick start guides and for folks testing multiple comparable solutions.
Do you have a workaround or are completely blocked by this? :
Workaround are solutions like that CNCF project that deploys Kubernetes on Docker (small scale); minikube and similar. But these are all pretty hefty in I/O usage.
Name of your Organization (Optional)
Any Other info.
Happy to engineer once the architecture has been agreed upon (assuming there is interest)
Does the interactive context in notebooks fill this need? And/or the local runner?
This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.
Update: been updating google/ml-metadata to latest Bazel (incl. Bzlmod) and its dependencies to support newer versions of Python; then going to contribute similarly to this repository for the same effect.
Have to agree with @rcrowe-google that the existing local-runner is perfectly sufficient for this.