knapply/datapan

datapan

This is still a worflow test bed. It is not yet useful.

datapan sifts through enormous files in parallelized Rust to only grab the data you want as quickly and memory-efficiently as possilbe.

Installation

## create/activate venv
# sudo apt-get install python3-venv
# python -m venv datapan_env
# source datapan_env/bin/activate
# python -m pip install --upgrade pip

pip install datapan

Usage

import datapan

some_dir = ""

test = datapan.hello_rust(some_dir)

print(test)

Developer Version

• Rust (nightly)

curl https://sh.rustup.rs -sSf | sh
# rustup default nightly
rustup update nightly

• Poetry

pip install poetry

make install
make test