This package contains common utility functions for usual tasks at Wellcome Data Labs. In particular:
modules | description |
---|---|
io | manipulating data, in and out S3, and processing |
ml | wrappers for processing texts, vectorisers and classifiers |
spacy | common utils for converting data form and to spacy/prodigy format |
mis/viz | any other utils, including Wellcome colour palletes |
For more in depth information see the /examples
folder and release notes.
Installing from PyPi
pip install wellcomeml
This will install the "vanilla" package. In order to install the deep-learning functionality (torch/transformers/spacy transformers):
pip install wellcomeml[deep-learning]
Installing from a release wheel
Download the wheel from aws and pip install it:
pip install wellcomeml-2020.1.0-py3-none-any.whl
pip install wellcomeml-2020.1.0-py3-none-any.whl[deep-learning]
make
Create a github token with artifact write access and export it to the env variables:
export GITHUB_TOKEN=...
After making changes, in order to buil a new wheel, run:
make dist
pip3 install <relative path to this folder>
Some experimental features (currently wellcomeml.ml.SemanticEquivalenceClassifier
) require a version of transformers
that is not compatible with spacy-transformers
. To develop those features:
export WELLCOMEML_ENV=development_transformers
pip install -r requirements_transformers.txt --upgrade
On OSX, ff you get a message complaining about the rust compiler, install and initialise it with:
brew install rustup
rustup-init
Examples can be found in the subfolder examples
.