SIGABRT on interface detach
Fred-si opened this issue · 6 comments
Hello. I work on project that use SQLAlchemy+firebird. We have recently migrate from fdb to firebird-driver python package and we experience some python crashes with SIGABRT in our test suite.
These crashes happen after introduce little modification on code/test that seem to have no relation with test that crash. Sometime, the modified code is not executed before crash. Crashes can happen at different places in tests suite, during a database operation or not, but always at the place will code not change. They can happen in Gitlab runner or locally in venv.
Fatal Python error: Aborted
Current thread 0x00007fd5d3a7d000 (most recent call first):
Garbage-collecting
File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/interfaces.py", line 1179 in detach
File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1645 in __del__
File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1145 in __init__
File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1545 in __init__
File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1935 in info
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy_firebird/firebird.py", line 147 in _get_server_version_info
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 514 in initialize
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy_firebird/base.py", line 428 in initialize
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 732 in first_connect
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 1910 in go
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 487 in __call__
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 473 in _exec_w_sync_on_first_run
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 916 in __connect
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678 in __init__
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393 in _create_connection
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 167 in _do_get
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716 in checkout
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1269 in _checkout
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452 in connect
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3292 in raw_connection
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145 in __init__
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3268 in connect
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1143 in _connection_for_bind
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139 in _go
File "<string>", line 2 in _connection_for_bind
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2045 in _connection_for_bind
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2178 in _execute_internal
File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2306 in execute
File "/project_root/tests/utils/abstract_test_base.py", line 63 in _clean_database
File "/project_root/tests/utils/abstract_test_base.py", line 40 in setup_class
File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 777 in _call_with_optional_argument
File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 856 in xunit_setup_class_fixture
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 895 in call_fixture_func
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1123 in pytest_fixture_setup
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1069 in execute
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 693 in _compute_fixture_value
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 607 in _get_active_fixturedef
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 585 in getfixturevalue
File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 566 in _fillfixtures
File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 1795 in setup
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 494 in setup
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 157 in pytest_runtest_setup
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 127 in runtestprotocol
File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/project_root/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
File "/project_root/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
File "/project_root/venv/bin/pytest", line 8 in <module>
Extension modules: _cffi_backend, charset_normalizer.md, simplejson._speedups, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, yaml._yaml, markupsafe._speedups, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, dependency_injector.providers, dependency_injector._cwiring, dependency_injector.containers, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, google._upb._message (total: 97)
zsh: IOT instruction (core dumped) pytest
Fatal Python error: Aborted
Current thread 0x00007f509adb9b80 (most recent call first):
Garbage-collecting
File "/usr/local/lib/python3.11/site-packages/firebird/driver/interfaces.py", line 1179 in detach
File "/usr/local/lib/python3.11/site-packages/firebird/driver/core.py", line 1645 in __del__
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2167 in lr0_goto
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2334 in reads_relation
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2454 in <lambda>
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2074 in traverse
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2065 in digraph
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2455 in compute_read_sets
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2518 in add_lalr_lookaheads
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2551 in lr_parse_table
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2135 in __init__
File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 3438 in yacc
File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 47 in parse_token_stream
File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 32 in parse
File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 14 in parse
File "/builds/a6cmo/agde6_support/shared/services/bodacc/helpers.py", line 12 in nested_get
File "/builds/a6cmo/agde6_support/shared/services/bodacc/pcl_service.py", line 160 in _create_evt_jur
File "/builds/a6cmo/agde6_support/shared/services/bodacc/pcl_service.py", line 89 in _import_data
File "/builds/a6cmo/agde6_support/shared/services/bodacc/bodacc_service.py", line 210 in import_files
File "/builds/a6cmo/agde6_support/tests/services/bodacc/test_pcl_service.py", line 343 in test_import_creation_evt_existe_deja
File "/usr/local/lib/python3.11/unittest/mock.py", line 1375 in patched
File "/usr/local/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/usr/local/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
File "/usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
File "/usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
File "/usr/local/bin/pytest", line 8 in <module>
Extension modules: _cffi_backend, charset_normalizer.md, simplejson._speedups, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, yaml._yaml, markupsafe._speedups, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, dependency_injector.providers, dependency_injector._cwiring, dependency_injector.containers, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, google._upb._message (total: 97)
/usr/bin/bash: line 151: 18 Aborted (core dumped) pytest -p no:warnings --cov --cov-report term --cov-report html
The common points on all crashes is that happen on detach
method of iAttachment_v3
on interface.py
.
def detach(self) -> None:
"Replaces `isc_detach_database()`. On success releases interface."
self.vtable.deprecatedDetach(self, self.status)
self._check()
self._refcnt -= 1
Adding if condition before vtable.deprecatedDetach
seems to solve the problem, but I have no experience on database driver development, I don't know if it’s good idea.
def detach(self) -> None:
"Replaces `isc_detach_database()`. On success releases interface."
if self._refcnt:
self.vtable.deprecatedDetach(self, self.status)
self._check()
self._refcnt -= 1
Someone might be able to help us solve this problem properly?
First, could you please provide information about: firebird-driver version, Python version and Firebird version?
Second, the error happens in __del__
destructor. This is always very tricky to debug as the condition when this code is executed is not deterministic (hence it's not easy if even possible to create reliable reproducible test case). The general recommendation is to always close resources such as Connection explicitly (or use it as context manager) to avoid situations when object is released by memory manager.
But in this case, it could be something more subtle, as the culprit seems to be the broken interface reference counting (which couldn't happen in normal circumstances).
Hello
project run on python 3.11, firebird-driver is in 1.10.0 and SQLAlchemy in version 2.0.22, the project use also sqlalchemy-firebird in version 2.0.2.
Firebird server is jacobalberty/firebird:v4.0.2 docker image.
Firebird client lib can be in 4.0.2 when project run in docker or 3.08 when used in venv (provided by ubuntu firebird package).
firebird-driver v1.8.0 does not seem to be affected, but I’m not sure.
firebird-driver v1.9.0 and v1.8.0 are also affected
I see. I think that problem could come from use of 3.0.8 Firebird client library to access the v4.0.2 server.
Sure, this should be still considered as bug (although probably not in firebird-driver, I don't see how driver could be a culprit here), as this shouldn't happen even if older client is used to access newer server. But these cross-version scenarios are not part of testing (neither Firebird QA or firebird-driver), so it's possible that some backward compatibility issues could get unnoticed.
Could you please verify whether this issue goes away with 4.0.2 client lib vs. 4.0.2 server?
No, issue still occurs when using 4.0.2 client lib with 4.0.2 server.
Interesting. What really puzzles me is that it fails in deprecatedDetach
i.e. detach
in IAttachment version 3. This interface was changed to version 4 and then to 5 where detach
and dropDatabase
are redefined (hence the old detach
become deprecatedDetach
). This change to new interface version happened in 3.0.7 => 3.0.8, 4.0.0 => 4.0.1, so neither your version 4.0.2 or 3.0.8 should use this old interface version.
Anyway, the Connection.__del__
is a tricky part, so your proposed fix is good as any other, so I'll adopt it.
Ok, thanks a lot for your help 🙏