Async calls do synchronous sleeping while waiting for predictions.

Question

Async calls do synchronous sleeping while waiting for predictions.

LucasSwitz opened this issue 6 months ago · 3 comments

It looks like the replicate client is synchronously waiting for predictions to finish during async methods.

replicate_client.async_run eventually calls this function that waits on the prediction: https://github.com/replicate/replicate-python/blob/main/replicate/run.py

prediction.wait() looks to be a synchronously spinning function: https://github.com/replicate/replicate-python/blob/main/replicate/prediction.py#L122

Am I understanding this correctly? Can we add a prediction.async_wait() that uses asyncio.wait() instead of time.sleep().

Answer 1 · 2024-01-23T04:14:22.000Z

#225 fixes this partly, wanted to note that prediction.output_iterator() still does a blocking sleep: https://github.com/replicate/replicate-python/blob/a3784c43503fa4f1193b7b5a1c84bca662088c6d/replicate/prediction.py#L200

That is called from _make_output_iterator in async_run.

Answer 2 · 2024-01-23T22:18:29.000Z

Hi @LucasSwitz and @rohan-mehta. Thanks for reporting this. I just opened #230, which should address this. Please take a look and let me know if that works as expected.

Answer 3 · 2024-01-27T20:30:08.000Z

This is now available in version 0.23.1.