Error: "Connection pool is full"
andreasjansson opened this issue · 1 comments
andreasjansson commented
When parallelizing requests with asyncio (wrapping replicate.run in asyncio.get_running_loop().run_in_executor
I get
WARNING - Connection pool is full, discarding connection: api.replicate.com. Connection pool size: 10
Having an async client might fix this (see #74).
mattt commented
Hi @andreasjansson. Async support for the Replicate client library just landed in version 0.18.0. When you have an opportunity, can you try replacing run
with async_run
in your code, and let me know whether that resolved your issue?
Here's an example of how to run multiple predictions concurrently and wait for results:
import asyncio
import replicate
# https://replicate.com/stability-ai/sdxl
model_version = "stability-ai/sdxl:39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"
prompts = [
f"A chariot pulled by a team of {count} rainbow unicorns"
for count in ["two", "four", "six", "eight"]
]
async with asyncio.TaskGroup() as tg:
tasks = [
tg.create_task(replicate.async_run(model_version, input={"prompt": prompt}))
for prompt in prompts
]
results = await asyncio.gather(*tasks)
print(results)