PgStac no longer works with Azure Flexible Postgres server
martys opened this issue · 15 comments
A few days ago Microsoft has bumped up a minor version of a "Azure Database for PostgreSQL flexible server" to version 13.12
After that all PGStac methods using that are utilized by Stac-Fast-Api stopped working !
resulting in a "segmentation fault" and not returning any data back.
Executing the same queries or stored procedures using a PGAdmin works fine, indicating a problem with pgstab library
We are in the middle of production outage right now due to this, and will add more details as they become available.
here are some more detailed log's from our code
Traceback (most recent call last):
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 218, in release
self._con.terminate()
AttributeError: 'NoneType' object has no attribute 'terminate'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
result = await app(self.scope, self.receive, self.send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
return await self.app(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/applications.py", line 261, in __call__
await super().__call__(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__
await self.middleware_stack(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__
raise exc
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__
await self.app(scope, receive, _send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/api/middleware.py", line 76, in __call__
await self.app(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
await self.app(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/brotli_asgi/__init__.py", line 81, in __call__
await br_responder(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/brotli_asgi/__init__.py", line 126, in __call__
await self.app(scope, receive, self.send_with_brotli)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
raise exc
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__
await self.app(scope, receive, sender)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
raise e
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 656, in __call__
await route.handle(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 259, in handle
await self.app(scope, receive, send)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 61, in app
response = await func(request)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
raw_response = await run_endpoint_function(
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
return await dependant.call(**values)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/api/routes.py", line 57, in _endpoint
await func(request=request, **request_data.kwargs()), response_class
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/pgstac/core.py", line 42, in all_collections
async with pool.acquire() as conn:
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 1012, in __aexit__
await self.pool.release(con)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 886, in release
return await asyncio.shield(ch.release(timeout))
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 220, in release
raise ex
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 210, in release
await self._con.reset(timeout=budget)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/connection.py", line 1366, in reset
await self.execute(reset_query, timeout=timeout)
File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/connection.py", line 317, in execute
return await self._protocol.query(query, timeout)
File "asyncpg/protocol/protocol.pyx", line 338, in query
asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation
like 1
same issue with v14 or v15 as well
@martys the error messages you are receiving do not look like it is a Postgres version issue, it looks like there is an issue with getting your connection using asyncpg. Can you confirm that you can use the asyncpg library to make a connection from the machine you are running stac_fastapi on to your database?
Did you change your VPC configuration in Azure at all? Did Azure Flex Server change to requiring SSL or anything like that?
@bitner - no we don't recall doing any changes. Just noticed that Azure has executed and upgrade from 13.11 to 13.12
We've deployed a 13.12 server locally and the code works - so this might be azure specific. Just not sure how to get to the bottom of the issue.
Just as an additional datapoint, we are running pgstac 0.7.5 on Azure Postgres Flexible Server with PostgreSQL 14.8 and haven't noticed any connectivity issues.
@martys I would definitely start by just try connecting to your Azure Flex Server using asyncpg and just running any query ie SELECT 1; from the server you are running stac-fastapi on. Perhaps they updated a certificate or something for SSL?
SELECT 1; works
That error still looks like it is related to the database connection - I don't see any evidence in that traceback that indicates that indicates that any pgstac queries have even made it to the database. That error is coming from the asyncpg library, but is not indicating any kind of bad query - it is indicating some kind of issue with your connection.
When you tried SELECT 1; did you run that using asyncpg with the same connection parameters from the same server that you are having problems on with stac-fastapi-pgstac?
we're suspecting the connection pooling at this point, but still working to reproduce exactly
@martys I have no idea how to help you with this. I have no capability to reproduce this.
here's a sample project - with a public database in the cloud made available that easily reproduces the issue,
which seems to happen after the data is read from the DB and the connection is being disposed
import asyncio
from stac_fastapi.pgstac.db import DB
from stac_fastapi.pgstac.config import Settings
def create_db():
db = DB()
settings = Settings(
postgres_dbname = 'postgres',
postgres_user = 'demoadmin',
postgres_pass = 'vPf3BtYM1WzvbAJ',
postgres_host_reader = 'testpgstac.postgres.database.azure.com',
postgres_host_writer = 'testpgstac.postgres.database.azure.com',
postgres_port = '5432'
)
return db,settings
async def run():
db, settings = create_db()
readpool = settings.reader_connection_string
pool = await db.create_pool(readpool, settings)
async with pool.acquire() as conn:
collections = await conn.fetchval(
"""
SELECT * FROM all_collections();
"""
)
print(collections)
print("Done")
loop = asyncio.get_event_loop()
loop.run_until_complete(run())
Closing as can't reproduce.
Issue resolved from Microsoft side.
Thanks to everyone involved!