Whether the pre- and post-processing operations of batch processing are parallel

[pengxin233]

📚 The doc issue

During batch processing, torchserve accumulates the number of corresponding batches. When preprocessing, are there parameters that can control torchserve to perform preprocessing in parallel, and then perform inference together? Or do I need to implement parallel logic in the handle myself?

Suggest a potential alternative/fix

No response

[agunapal]

@pengxin233 you can use microbatching in TorchServe to do this

https://github.com/pytorch/serve/tree/master/examples/micro_batching