Blueprints and import into gunicorn.conf.py results in gunicorn reload bug

[mkmoisen]

This is a very interesting bug.

When using Blueprints with flask, and importing something into gunicorn.conf.py from the same top level module in which the flask application resides, Gunicorn's --reload functionality breaks. Specifically, the reloading functionality appears to work- gunicorn informs us that the application has been reloaded; but the run time code is actually cached, and the application in effect does not reload.

This bug is an issue because it makes development inconvenient, and it is time consuming to isolate.

This bug cannot be replicated if:

• you don't use blueprints,
• or if you use blueprints but do not import into gunicorn.conf.py from a module that exists inside the application module
• or if you use blueprints, and lazy import into gunicorn.conf.py

The reason I believe this is a flask bug, instead of a gunicorn bug, is that this bug only exists when using blueprints.

I've replicated this on Python 3.11.5, Flask 2.3.3 and 2.0.0, Gunicorn 21.2.0 and 21.0.0, and RHEL and Ubuntu.

Docker

I've uploaded a repo with a Dockerfile and docker-compose.yaml that can be used to quickly reproduce the bug. The application runs on port 8080.

git clone https://github.com/mkmoisen/blueprintbug.git
docker compose up --build

Directory Structure

blueprintbug/
    app/
        __init__.py
    gunicorn.conf.py
    requirements.txt

blueprintbug/app/init.py

from flask import Flask, Blueprint

def init_application():
    print("Application initialized.")


def cleanup_application():
    print("Application cleaned up.")


def create_app():
    """
    Creates & initializes a Flask application instance.
    :return:
    """
    app = Flask(__name__, static_folder=None, template_folder=None)

    app.register_blueprint(bp)

    return app


bp = Blueprint('home', __name__)

@bp.get('/')
def home():
    raise Exception("foo")

blueprintbug/gunicorn.conf.py

# The root cause of this bug is these imports
# Moving these imports to inside of post_worker_init and worker_exit eliminates the bug
from app import init_application
from app import cleanup_application

bind = '0.0.0.0:8080'

preload_app = False

worker_class = 'gthread'
workers = 1
threads = 1


def post_worker_init(worker):
    init_application()


def worker_exit(server, worker):
    cleanup_application()

blueprintbug/requirements.txt

Flask==2.3.3
gunicorn==21.2.0

Running the application

Run the application with the reload option:

gunicorn -c gunicorn.conf.py "app:create_app()" --reload

With the above code, if we call GET /, we can see the following exception, which we expect:

curl http://localhost:8080/

blueprintbug  |   File "/usr/src/app/app/__init__.py", line 27, in home
blueprintbug  |     raise Exception("foo")
blueprintbug  | Exception: foo

Now let's change blueprintbug/app/__init__.py to raise bar instead of foo:

@bp.get('/')
def home():
    raise Exception("bar")

And we can confirm that gunicorn appears to have reloaded the application:

blueprintbug  | [2023-09-20 17:08:29 +0000] [7] [INFO] Worker reloading: /usr/src/app/app/__init__.py modified
blueprintbug  | [2023-09-20 17:08:30 +0000] [7] [INFO] Worker exiting (pid: 7)
blueprintbug  | Application initialized.
blueprintbug  | Application cleaned up.
blueprintbug  | [2023-09-20 17:08:30 +0000] [10] [INFO] Booting worker with pid: 10

Next we can call GET / which will result in this strange result:

curl http://localhost:8080/

blueprintbug  |   File "/usr/src/app/app/__init__.py", line 27, in home
blueprintbug  |     raise Exception("bar")
blueprintbug  | Exception: foo

Notice how the stack trace says raise Exception("bar"), but how the actual run time exception raised is foo.

This implies that gunicorn is caching the run time code, even though it recognizes that the code has changed.

We would have expected the following console output, which does not happen:

blueprintbug  |   File "/usr/src/app/app/__init__.py", line 27, in home
blueprintbug  |     raise Exception("bar")
blueprintbug  | Exception: bar

---

The quickest workaround appears to be using lazy loading inside of gunicorn.conf.py.

# Do not import here, instead import inside the functions

def post_worker_init(worker):
    from app import init_application
    init_application()


def worker_exit(server, worker):
    from app import cleanup_application
    cleanup_application()

Alternatively, you can move the init_application and cleanup_application to a separate directory from /app.

Or you can not use blueprints.