The idea is basically orchestrate match between queries and corpus datasets
Declare any dependencies in src/requirements.txt for pip installation and src/environment.yml for conda installation.
To install them, run:
pip install -r src/requirements.txt
You can run your Kedro project with:
kedro run
Using the Data Catalog orchestration we can separate search and base data to create our sentence pairing.
Queries: data/01_raw/queries.csv
Corpus: data/01_raw/corpus.csv
They are adapted to create match between two csv simple files. Just put the data you want and try to run the kedro pipeline with CUDA + sentece-transformers.
conf/base/parameters.yml
Check sentence-transformers and kedro documentation:
