Post request response time spikes

[chmjelek]

Hello,

in my project I need to upload images as bytes to an endpoint and then process them, within a certain time. The images have a resolution of 600x600 and weight ~700 kB per image. Images are in png format. I have created a minimal flask application with one endpoint. The application is served using a waitress.

The problem is that sometimes response time from an endpoint suddenly increases by an order of magnitude. As I have to fit in a certain amount of time, such spikes can cause a critical error.

I ran a cProfile on the post request and found out that when there is a spike two functions take more time than normal:

Normal

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.000    0.000    0.000    0.000 {method 'sendall' of '_socket.socket' objects}
     1    0.004    0.004    0.004    0.004 {method 'recv_into' of '_socket.socket' objects}

Spike

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.008    0.004    0.008    0.004 {method 'sendall' of '_socket.socket' objects}
     1    0.012    0.012    0.012    0.012 {method 'recv_into' of '_socket.socket' objects}

What is the cause of these spikes, and can they be resolved in some way?

Code to reproduce:

# app.py
import numpy as np
from flask import Flask, request


def create_app():
    app = Flask(__name__)

    @app.route("/endpoint")
    def endpoint():
        data = request.data
        img_arr = np.frombuffer(data, dtype=np.uint8)
        # ...
        return {"foo": "bar"}

    return app

# wsgi.py
from waitress import serve

from app import create_app


def main():
    app = create_app()
    serve(app, host="0.0.0.0", port="9009")


if __name__ == "__main__":
    main()

# send.py

from pathlib import Path

import numpy as np
import requests
from PIL import Image


def main():
    img_paths = list(Path("images").rglob("*.png"))
    ses = requests.Session()
    speeds = []
    for img_path in img_paths:
        img_arr = np.asarray(Image.open(img_path))
        response = ses.post(url='http://0.0.0.0:9009/endpoint', data=img_arr.tobytes())
        speeds.append(response.elapsed.total_seconds())


if __name__ == "__main__":
    main()

Python and package versions:

python==3.8.18

Flask==3.0.2
numpy==1.24.4
pillow==10.2.0
requests==2.31.0
waitress==3.0.0

Three independent performances for 1,000 images:

First

Second

Third

[davidism]

Sorry, this is outside the scope of what we can help with here. It could be any number of things including the network, OS, filesystem, HTTP server, a proxy, other applications, etc. If you track this down to a specific issue in Flask, we'll be happy to reconsider it.