Better non blocking
hadim opened this issue · 6 comments
The current version uses async features and indeed, downloading is done on-the-fly.
But since the code is running in the main thread and is very resource-intensive, JLab UI often blocks. The easiest example is trying to navigate in folders during a download of an archive.
A solution would be to run archive_and_download() in another thread using ioloop.IOLoop.current().run_in_executor. But this is not compatible with the call self.flush inside archive_and_download() to send the data over the network. So far I haven't found a good solution.
Any ideas are welcome.
Hey @hadim, I think compressing the data to a buffer using run_in_executor asynchronously. And then flushing that buffer may work.
But as I said flushing cannot be done in run_in_executor
Sorry I wasn't clear enough. We could use a buffer like IOBytes. Then once the compression is done, the buffer is flushed to the handler. The drawback is that we will consume server in memory to store the buffer.
So you mean storing the whole archive file in a buffer like IOBytes? In that case, it will be impossible to archive and download folder bigger than the available memory on the server. The whole point of doing on-the-fly archiving and downloading was to allow this.
Unless there is something I didn't understand?
What we need is to do all the heavy work in an ioloop.IOLoop.current().run_in_executor. And inside that executor call sef.flush inside the main thread. But I don't know how to call a function in the main thread within an executor.