achyudh/tardis

Pickle inputs for faster loading times

Closed this issue · 1 comments

Is your feature request related to a problem? Please describe.
Since we pre-process the inputs every time we run the script, it takes around an hour to load the data and pre-process it

Describe the solution you'd like
Pickle all inputs after pre-processing and store it to disk. Add data/cache module to load from disk

Describe alternatives you've considered
We considered multi-processing for pre-processing the input in parallel, but this doesn't result in considerably lower loading times due to single threaded bottlenecks

Additional context
Add any other context or screenshots about the feature request here.

Resolved in karkaroff#9