data loader part
avijit9 opened this issue · 7 comments
Is it possible to provide the code for the dataloader part? I have implemented something, but it is taking around 70 secs to fetch one batch of data.
You might like to try torchvideo's gulp dataset: https://github.com/torchvideo/torchvideo
You can use this with the standard pytorch dataloader.
A few other tips: make sure you data is on fast storage, preferably a local SSD, failing that a local HDD. Network storage has horrendous random access speeds.
Thanks for your reply. Does torchvideo support optical flow?
What is the usual time required for loading a batch data using the EpicVideoDataset for you?
Does torchvideo support optical flow?
Not currently.
What is the usual time required for loading a batch data using the EpicVideoDataset for you?
Depends on where the data is being loaded from, how many frames are being sampled and how big the batch size is. But on a beefy machine a 128 batch size with 20 workers loads in about .4s with heavy data augmentation
Is it possible to share the data loading pipeline? I am running my code on a server which has quite a good configuration. But it still takes a lot of time (around ~100 secs/batch). Not sure what is going wrong. I have set the batch_size=16
Creating a data loading pipeline is nontrivial, but there are many resources online you can rely on. There's nothing specific about our dataset in terms of data loading, so we are not planning to release a data loading pipeline specifically.
We are particularly busy with creating the leaderboards and evaluation code at the moment.
We will consider doing this in the future.
Hi,
I just figured out the source of the problem I was facing. I somehow overlooked the indices argument in the load_frames method ofEpicVideoDataset. As a result, I was reading all the frames of a segment to sample some of them for training.
Thanks for your replies. Sorry for the trouble. :)