dynamic-batcher

dynamic_batcher is designed for inferencing DL models using GPU and enforces model's concurrency.

Installation

pip install dynamic_batcher

Quickstart

Additional Requirements

pip install -r requirements-test.txt

Run

redis RUN:

docker run --rm -p 6379:6379 -e ALLOW_EMPTY_PASSWORD=yes bitnami/redis:latest

app
ENV:

REDIS__HOST=localhost
REDIS__PORT=6379

RUN:

gunicorn e2e.app.main:app \
  -k=uvicorn.workers.UvicornWorker \
  --workers=4

batcher
ENV:

REDIS__HOST=localhost
REDIS__PORT=6379

DYNAMIC_BATCHER__BATCH_SIZE=64
DYNAMIC_BATCHER__BATCH_TIME=2

RUN:

python e2e/batcher/run.py

locust
RUN:
```
locust -f e2e/locust/locustfile.py
```

Test

swagger: http://localhost:8000

POST `/items/batch/{item_id}

curl -X POST http://localhost:8000/items/batch/1 \
  -H 'Content-Type: application/json' \
  -d '{
    "content": "string",
    "nested": {
      "key": "string",
      "values": [
        1,
        5,
        2
      ]
    }
  }'

{
  "data": {
    "content": "string",
    "nested": {
      "key": "string",
      "values": [
        1,
        5,
        2
      ],
      "result": [
        2,
        6,
        3
      ]
    },
    "name": "b0878740-47a8-4dd7-bfe8-9c5ed1fee4ea"
  },
  "elapsed_time": 2.551218032836914
}

locust: http://localhost:8089

Explanation

when DYNAMIC_BATCHER__BATCH_SIZE=64 and DYNAMIC_BATCHER__BATCH_TIME=2 is set,
a running BatchProcessor waits to run a batch until the amount of requests received is met(requests count=64), for the batch_time(2 seconds). If the time is up, the partial amount of requests will be processed.

Launch

start test daemon
BatchProcessor start: delay=0.001, batch_size=64, batch_time=2

Single request(concurrency=1)

batch start: 2.001/2, 1/64
batch start: 2,001/2, 1/64
...

Concurrent requests(concurrency=100)

batch start: 1.653/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
batch start: 0.064/2, 64/64
...
batch start: 2.001/2, 36/64

Concept

ref:

https://github.com/triton-inference-server/tutorials/tree/main/Conceptual_Guide/Part_2-improving_resource_utilization#what-is-dynamic-batching

pydemia/dynamic-batcher