ml6team/fondant

Cleanly split or remove component and pipeline SDKs / extras

Opened this issue · 0 comments

With the introduction of the lightweight components, our component and pipeline SDKs are no longer cleanly split. We should either split them again, or include the optional installs for both in the default installation.

This is the error I currently get when trying to run a pipeline without any extras installed. The pipeline does not use a lightweight component.

(base) jupyter@download-images-small-eu:~/src$ fondant run local pipeline.py
Traceback (most recent call last):
File "/opt/conda/bin/fondant", line 5, in
from fondant.cli import entrypoint
File "/opt/conda/lib/python3.10/site-packages/fondant/cli.py", line 32, in
from fondant.pipeline import Pipeline
File "/opt/conda/lib/python3.10/site-packages/fondant/pipeline/init.py", line 1, in
from .lightweight_component import ( # noqa
File "/opt/conda/lib/python3.10/site-packages/fondant/pipeline/lightweight_component.py", line 13, in
from fondant.component import BaseComponent, Component
File "/opt/conda/lib/python3.10/site-packages/fondant/component/init.py", line 12, in
from .component import ( # noqa
File "/opt/conda/lib/python3.10/site-packages/fondant/component/component.py", line 7, in
import dask
ModuleNotFoundError: No module named 'dask'