pytorch/data

Improve AsyncIODataPipe

Opened this issue · 0 comments

ejguan commented

🚀 The feature

A few improvements can be achieved for AsyncIODataPipe:

  • Make it working properly with nested_async
  • Constantly fetching rather than fetching batch per batch
    with closing(nested_async.prefetch_sequence(datapipe, self.max_prefetch)) as seq:
  • Align the API with threadpool_map without the need to specify batch.

Motivation, pitch

Improve performance, reliability, functionality.

Alternatives

No response

Additional context

No response