Pickle inputs for faster loading times
Closed this issue · 1 comments
achyudh commented
Is your feature request related to a problem? Please describe.
Since we pre-process the inputs every time we run the script, it takes around an hour to load the data and pre-process it
Describe the solution you'd like
Pickle all inputs after pre-processing and store it to disk. Add data/cache module to load from disk
Describe alternatives you've considered
We considered multi-processing for pre-processing the input in parallel, but this doesn't result in considerably lower loading times due to single threaded bottlenecks
Additional context
Add any other context or screenshots about the feature request here.
achyudh commented
Resolved in karkaroff#9