GZipMiddleware cannot be used with response validation
sgranel-fsk opened this issue · 2 comments
sgranel-fsk commented
Description
It is not possible to activate response validation (validate_responses parameter of the Flask app), and use the GZipMiddleware suggested here https://connexion.readthedocs.io/en/latest/middleware.html#list-of-useful-middleware.
As it is not possible to insert a middleware after response validation (or I didn't find it), the response validation tries to decode binary data as utf-8. Content-encoding header does not seems to be considered.
Expected behaviour
Be able to activate both GZipMiddleware and response validation.
Actual behaviour
Internal server error is raised:
[27/Mar/2024:16:31:43 +0100] - PFSK00081 - forsk.explorer - connexion.middleware.exceptions - ERROR - n/a -- UnicodeDecodeError('utf-8', b'\x1f\x8b\x08\x00\xdc;\x04f\x02\xff\xed\xcd\xb1j\xc30\x14@\xd1\xbd_a4\x87\xe2\x86\x90\xa1k\xa1s\xf7\x92\xc1$J1\xb8\x96Q\xe5!\x18\xff{m\x87.]\xf2\x03g{\xe8]\xbd3\x85r\x1bbx\xad\xc2{l\xca\x98\xe3[\xea\xbax.m\xea\xc3\xae\n\xd7\xfb\xe3\xcf\x12|N\xff\xd35\x18r\x1ab.\xed\x96L\xa1\xe9J[\xc6\xcbZ\xf5c\xd7\xcdK\xf1\x15\xd3w,\xf9\xb6\xed\xff.|\xa4\xb6/\xeb\xffsJ\xf9\xd2\xf6M\xb9\x1b\x87\x97\xe3\xbe\xae\x9f\xeb]u<\xec\xebm<\xcd\xcb\x156\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9l6\x9b\xcdf\xb3\xd9\xec\xc7\xf6i~\xfa\x05P\x95\xa9\xda\x8c\xe4\x01\x00', 1, 2, 'invalid start byte')
Traceback (most recent call last):
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/swagger_ui.py", line 222, in __call__
await self.router(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 487, in handle
await self.app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 808, in app
await self.default(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/swagger_ui.py", line 235, in default_fn
await self.app(original_scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/routing.py", line 154, in __call__
await self.router(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 487, in handle
await self.app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
await self.app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/routing.py", line 48, in __call__
await self.next_app(original_scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/security.py", line 111, in __call__
await self.next_app(scope, receive, send)
File "/home/GIT/naos-service-commons/naos_base_app/validators.py", line 107, in __call__
await self.app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/request_validation.py", line 142, in __call__
await self.next_app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/response_validation.py", line 127, in __call__
await self.next_app(scope, receive, wrapped_send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/lifespan.py", line 26, in __call__
await self.next_app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/abstract.py", line 264, in __call__
return await operation(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/context.py", line 25, in __call__
await self.next_app(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 24, in __call__
await responder(scope, receive, send)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 44, in __call__
await self.app(scope, receive, self.send_with_gzip)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/apps/flask.py", line 151, in __call__
return await self.asgi_app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/a2wsgi/wsgi.py", line 165, in __call__
return await responder(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/a2wsgi/wsgi.py", line 206, in __call__
await sender
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/a2wsgi/wsgi.py", line 226, in sender
await send(message)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/starlette/middleware/gzip.py", line 109, in send_with_gzip
await self.send(message)
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/middleware/response_validation.py", line 125, in wrapped_send
return await send(message)
^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/validators/abstract.py", line 201, in send_
body = self._parse(stream)
^^^^^^^^^^^^^^^^^^^
File "/home/GIT/naos-explorer/.venv_connexion3/lib/python3.11/site-packages/connexion/validators/json.py", line 117, in _parse
body = b"".join(stream).decode(self._encoding)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Steps to reproduce
Create a Flask app with the validate_responses activated and the starlette GZipMiddleware :
default_middleware = ConnexionMiddleware.default_middlewares
default_middleware.append(GZipMiddleware)
connexion_app = connexion.FlaskApp('test', specification_dir='./specs',
middlewares=default_middleware,
validate_responses=True))
The route need to send a large response for the zip to be triggered, and you will get a 500 internal server error.
Additional info:
Seems similar to #1860 bur for response validation.
Output of the commands:
python --version
Python 3.11.5pip show connexion | grep "^Version\:"
Version: 3.0.6
RobbeSneyders commented
Hi @sgranel-fsk, can you try to insert the GZIPMiddleware in front of the ValidationMiddleware? As noted in the documentation, "responses hit the middlewares in reversed order.".
sgranel-fsk commented
Hi, thank you for the hint it is working !
Sorry I missed this point in documentation, I am closing this issue.