Asynchronous PyTorch model serving on top of event streaming platforms such as Apache Kafka or Google PubSub
-
Git clone (or download and unpack) the repository
-
Change to the project directory
$ cd AsyncTorchServe-Kafka-Pubsub/
- Setup and activate a Python virtual environment
$ python3 -m venv venv
$ source venv/bin/activate
- Install the requirements
$ python -m pip install --upgrade pip
$ python -m pip install --upgrade setuptools
$ python -m pip install --upgrade wheel
$ pip install -r requirements.txt
- Setup the configuration in
./config.ini
[options]
stream_broker = Kafka
; stream_broker = PubSub
[models] ; list of models to instantiate in the server
image_classification
[Kafka]
bootstrap_servers = localhost:9092
[PubSub]
project = ; enter google cloud project
account_keypath = ; enter path to the private key json
- For Apache Kafka, start the docker container:
$ docker-compose up
For Google PubSub,
install
the Google Cloud SDK gcloud
and setup a Google Cloud project
with PubSub API enabled.
- Add our package to Python's path before running the following scripts
$ export PYTHONPATH=./
- Create the streaming topics
$ python bin/topics.py --create
- Start the server
$ python async_torchserve/app.py --config=config.ini
- Send downloaded FashionMNIST images to the input topic interactively
$ python bin/push_data.py --download
- Pull the streaming predictions from the output topic
python bin/pull_data.py
- Cleanup by deleting the topics, especially if using Google PubSub
python bin/topics.py --delete