This a dataloader that for multiple training jobs in deep learning. The goal of this project is to avoid redundant data prep work and boost training efficiency.
This is an example of how you may give instructions on setting up your project locally. To get a local copy up and running follow these simple example steps.
Install packages for server
- Install the opencv-rust
- Install the tensorflow-rust
Install packages for client
pip install -r requirements.txt Build the target of server
cargo build --release- Run the server
./server/target/release/joader- Create a dataset with some keys and conditions
from dataset.dataset import Dataset as JDataset, DatasetType
channel = grpc.insecure_channel('127.0.0.1:4321')
ds = JDataset(name=name, location=location, ty=DatasetType.LMDB)
for k in keys:
ds.add_item([str(k).encode()])
ds.create(channel)
channel.close()- Register the job for loading data and read data
job = Job.new(dataset_name, name='job', ip='127.0.0.1:4321')
for _ in range(dataset_len):
data = job.next- Train the model with PyTorch
More examples are in client/test and unitests are in each file in server
Distributed under the MIT License. See LICENSE.txt for more information.