munhouiani/petastorm-demo

A demo for petastorm and pyspark

Petastorm Demo

This repo demonstrates the data pipeline from pyspark to pytorch.

Use this repo with https://blog.munhou.com/2020/01/17/Data-Pipeline-From-Pyspark-to-Pytorch/

Create environment

conda env create -f environment.yml

Execution

Run main.py

python main.py