Reduce number of dependencies for taps vs targets
Closed this issue · 3 comments
Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/394
Originally created by @laurentS on 2022-05-23 10:34:58
Summary
The sdk depends on a number of libraries that are only used in the tap or target context. This adds weight to virtual envs, potential for dependency conflicts, security issues, etc...
While I don't have an immediate solution, it would make sense to work towards a way to split dependencies according to the use-case of the sdk.
Proposed benefits
- smaller code footprint (weight, security, conflicts...)
- faster venv installs -> happy devs 😄
Proposal details
The singer recommendation is to install each tap/target in its own virtual env to minimize potential for dependency conflicts, etc...
As an example, we run tap-github and tap-gitlab, each of which pulls in singer-sdk and transitively sqlalchemy (and its own dependencies). As these are taps, they make no use of sqlalchemy (and likely never will). sqlalchemy is interesting because it also requires a bunch of build dependencies as it contains C code that's built upon install.
I suspect there are similar dependencies that are not used by the target part of the sdk.
I'm wondering if it would be possible to break up the dependency list into something like deps-tap and deps-target so that non-development installations of the sdk fetch fewer dependencies.
Best reasons not to build
I'm conscious this is probably not trivial, and might lead to headaches by creating multiple "shades" of the sdk.
poetry has an extras mechanism, but I don't know if it can handle variants, unless dependencies are individually declared as extras, which may also make installation less straightforward.
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
Superseded by #2335