Cleanly split or remove component and pipeline SDKs / extras
Opened this issue · 0 comments
With the introduction of the lightweight components, our component
and pipeline
SDKs are no longer cleanly split. We should either split them again, or include the optional installs for both in the default installation.
This is the error I currently get when trying to run a pipeline without any extras installed. The pipeline does not use a lightweight component.
(base) jupyter@download-images-small-eu:~/src$ fondant run local pipeline.py
Traceback (most recent call last):
File "/opt/conda/bin/fondant", line 5, in
from fondant.cli import entrypoint
File "/opt/conda/lib/python3.10/site-packages/fondant/cli.py", line 32, in
from fondant.pipeline import Pipeline
File "/opt/conda/lib/python3.10/site-packages/fondant/pipeline/init.py", line 1, in
from .lightweight_component import ( # noqa
File "/opt/conda/lib/python3.10/site-packages/fondant/pipeline/lightweight_component.py", line 13, in
from fondant.component import BaseComponent, Component
File "/opt/conda/lib/python3.10/site-packages/fondant/component/init.py", line 12, in
from .component import ( # noqa
File "/opt/conda/lib/python3.10/site-packages/fondant/component/component.py", line 7, in
import dask
ModuleNotFoundError: No module named 'dask'