GoogleCloudPlatform/cloud-sql-python-connector

system.test_connector_object: test_connector_with_custom_loop failed

flaky-bot opened this issue · 2 comments

flaky-bot commented

Note: #583 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

commit: 2423e2e
buildURL: https://github.com/GoogleCloudPlatform/cloud-sql-python-connector/actions/runs/6662018631
status: failed

Test output
def test_connector_with_custom_loop() -> None:
        """Test that Connector can be used with custom loop in background thread."""
        # create new event loop and start it in thread
        loop = asyncio.new_event_loop()
        thread = Thread(target=loop.run_forever, daemon=True)
        thread.start()

    with Connector(loop=loop) as connector:
        pool = init_connection_engine(connector)





      with pool.connect() as conn:





tests/system/test_connector_object.py:146:




.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/engine/base.py:3268: in connect

return self._connection_cls(self)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/engine/base.py:145: in init

self._dbapi_connection = engine.raw_connection()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/engine/base.py:3292: in raw_connection

return self.pool.connect()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:452: in connect

return _ConnectionFairy._checkout(self)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:1269: in _checkout

fairy = _ConnectionRecord.checkout(pool)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:716: in checkout

rec = pool._do_get()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:170: in _do_get

self._dec_overflow()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:146: in exit

raise exc_value.with_traceback(exc_tb)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/impl.py:167: in _do_get

return self._create_connection()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:393: in _create_connection

return _ConnectionRecord(self)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:678: in init

self.__connect()

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:903: in __connect

pool.logger.debug("Error on connect(): %s", e)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py:146: in exit

raise exc_value.with_traceback(exc_tb)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:898: in __connect

self.dbapi_connection = connection = pool._invoke_creator(self)

.nox/system-3-8/lib/python3.8/site-packages/sqlalchemy/pool/base.py:365: in 

return lambda rec: creator_fn()

tests/system/test_connector_object.py:38: in getconn

conn = custom_connector.connect(

google/cloud/sql/connector/connector.py:163: in connect

return connect_task.result()

../../../hostedtoolcache/Python/3.8.18/x64/lib/python3.8/concurrent/futures/_base.py:444: in result

return self.__get_result()

../../../hostedtoolcache/Python/3.8.18/x64/lib/python3.8/concurrent/futures/_base.py:389: in __get_result

raise self._exception

google/cloud/sql/connector/connector.py:244: in connect_async

instance_data, ip_address = await instance.connect_info(ip_type)

google/cloud/sql/connector/instance.py:446: in connect_info

instance_data = await self._current

google/cloud/sql/connector/instance.py:389: in _refresh_task

refresh_data = await refresh_task

google/cloud/sql/connector/instance.py:326: in _perform_refresh

ephemeral_cert = await ephemeral_task

google/cloud/sql/connector/refresh_utils.py:201: in _get_ephemeral

resp = await client_session.post(

.nox/system-3-8/lib/python3.8/site-packages/aiohttp/client.py:669: in _request

resp.raise_for_status()




self = <ClientResponse(https://sqladmin.googleapis.com/sql/v1beta4/projects/cloud-sql-connector-testing/instances/mysql-proxy...ength': '1613', 'Date': 'Fri, 27 Oct 2023 02:07:34 GMT', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000')>


def raise_for_status(self) -> None:
    if not self.ok:
        # reason should always be not None for a started response
        assert self.reason is not None
        self.release()





      raise ClientResponseError(





            self.request_info,
            self.history,
            status=self.status,
            message=self.reason,
            headers=self.headers,
        )



E           aiohttp.client_exceptions.ClientResponseError: 502, message='Bad Gateway', url=URL('https://sqladmin.googleapis.com/sql/v1beta4/projects/cloud-sql-connector-testing/instances/mysql-proxy-testing:generateEphemeralCert')


.nox/system-3-8/lib/python3.8/site-packages/aiohttp/client_reqrep.py:1011: ClientResponseError

flaky-bot commented

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

When run at the same commit (2423e2e), this test passed in one build (https://github.com/GoogleCloudPlatform/cloud-sql-python-connector/actions/runs/6662018631) and failed in another build (https://github.com/GoogleCloudPlatform/cloud-sql-python-connector/actions/runs/6662018631).

jackwotherspoon commented

aiohttp error that i have not seen before, will monitor for it appearing again

aiohttp.client_exceptions.ClientResponseError: 502, message='Bad Gateway