Segmentation fault when connecting using SSL
rubendv opened this issue · 34 comments
tl;dr
This segfault only affects the psycopg wheel package: the workaround is to build psycopg from source, e.g. using:
$ pip install --no-binary psycopg2
or the entry in the requirements.txt file:
psycopg2>=2.7,<2.8 --no-binary psycopg2
Running the delete_database function in this snippet against our database causes a segmentation fault in psycopg2==2.7 and 2.7.1, but not in 2.6.2. The segmentation fault seems to happen inside OpenSSL.
Debugging revealed that SSL_library_init is being called twice, once by the Python ssl module and once by the libpq library that comes with psycopg2, where it segfaults. Not sure if this function is allowed to be called twice or not.
import contextlib
import psycopg2
from fabfile.lib import info, parse_uri
@contextlib.contextmanager
def create_connection(connection_uri):
connection_args = parse_uri(connection_uri)
conn = psycopg2.connect(
database='postgres', user=connection_args.username,
password=connection_args.password, host=connection_args.hostname,
port=connection_args.port if connection_args.port else 5432)
conn.set_isolation_level(0)
yield conn.cursor()
conn.close()
def delete_database(connection_uri, database):
hostname = parse_uri(connection_uri).hostname
info('deleting database {} on {}'.format(database, hostname))
with create_connection(connection_uri) as cur:
cur.execute('''SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = %s
AND pid <> pg_backend_pid()''', (database, ))
cur.execute('DROP DATABASE "{}"'.format(database))
The system openssl is 1.0.1f-1ubuntu2.22 on Ubuntu 14.04.
On psycopg2==2.6.2 this works fine.
On psycopg2==2.7 we get the following segmentation fault:
Program received signal SIGSEGV, Segmentation fault.
engine_unlocked_finish (e=e@entry=0x7ffff626bfa0 <update>, unlock_for_handlers=unlock_for_handlers@entry=1) at eng_init.c:92
92 eng_init.c: No such file or directory.
(gdb) bt
#0 engine_unlocked_finish (e=e@entry=0x7ffff626bfa0 <update>, unlock_for_handlers=unlock_for_handlers@entry=1) at eng_init.c:92
#1 0x00007ffff6251d69 in ENGINE_finish (e=0x7ffff626bfa0 <update>) at eng_init.c:146
#2 0x00007ffff6264f05 in EVP_DigestInit_ex (ctx=0x10cb600, type=0x7ffff6540080 <sha1_md>, impl=0x0) at digest.c:162
#3 0x00007ffff28fe8a4 in ssl23_connect () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/./libssl-a65b360f.so.0.9.8e
#4 0x00007ffff2b63a9c in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#5 0x00007ffff2b63fbd in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#6 0x00007ffff2b5175e in PQconnectPoll () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#7 0x00007ffff2b520ce in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#8 0x00007ffff2b529bf in PQconnectdb () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-cf361c55.so.5.8
#9 0x00007ffff2d8c241 in _conn_sync_connect (self=0x7ffff2ff8c80) at psycopg/connection_int.c:716
#10 conn_connect (self=self@entry=0x7ffff2ff8c80, async=<optimized out>) at psycopg/connection_int.c:812
#11 0x00007ffff2d8cf84 in connection_setup (async=<optimized out>, dsn=<optimized out>, self=0x7ffff2ff8c80) at psycopg/connection_type.c:1277
#12 connection_init (obj=obj@entry=<psycopg2.extensions.connection at remote 0x7ffff2ff8c80>, args=args@entry=('dbname=postgres host=**censored ip** user=admin password=**censored password** port=5432',), kwds=kwds@entry=0x0)
at psycopg/connection_type.c:1362
#13 0x000000000055f6db in type_call.25495 (type=<optimized out>, type@entry=0x7ffff2fab460 <connectionType>, args=args@entry=('dbname=postgres host=**censored ip** user=admin password=**censored password** port=5432',), kwds=kwds@entry=0x0)
at ../Objects/typeobject.c:745
#14 0x00000000004c8a7d in PyObject_Call (kw=0x0, arg=('dbname=postgres host=**censored ip** user=admin password=**censored password** port=5432',), func=<type at remote 0x7ffff2fab460>) at ../Objects/abstract.c:2529
#15 call_function_tail.5417 (callable=callable@entry=<type at remote 0x7ffff2fab460>, args=('dbname=postgres host=**censored ip** user=admin password=**censored password** port=5432',)) at ../Objects/abstract.c:2561
#16 0x00000000005ab155 in _PyObject_CallFunction_SizeT (callable=<type at remote 0x7ffff2fab460>, format=format@entry=0x7ffff2da3f60 "s") at ../Objects/abstract.c:2605
#17 0x00007ffff2d860b8 in psyco_connect (self=<optimized out>, args=<optimized out>, keywds=<optimized out>) at psycopg/psycopgmodule.c:111
#18 0x000000000052714b in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffc340, func=<built-in function _connect>) at ../Python/ceval.c:4330
#19 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff3071850, for file /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py, line 130, in connect (dsn='dbname=postgres host=**censored ip** user=admin password=**censored password** port=5432', connection_factory=None, cursor_factory=None, kwargs={'port': 5432, 'host': '**censored ip**', 'password': '**censored password**', 'user': 'admin', 'database': 'postgres'}, kwasync={}), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#20 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff3047130, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=<optimized out>, kwcount=kwcount@entry=5,
defs=defs@entry=0x7ffff305b838, defcount=3, closure=0x0) at ../Python/ceval.c:3252
#21 0x0000000000524338 in fast_function (nk=5, na=0, n=<optimized out>, pp_stack=0x7fffffffc530, func=<function at remote 0x7ffff3061050>) at ../Python/ceval.c:4116
#22 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc530) at ../Python/ceval.c:4041
#23 PyEval_EvalFrameEx (
f=f@entry=Frame 0xf37790, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/postgresql.py, line 21, in create_connection (connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', connection_args=<ParseResult(database='') at remote 0x7ffff3061938>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#24 0x0000000000567b1e in gen_send_ex.isra.0 (exc=0, arg=0x0) at ../Objects/genobject.c:85
#25 gen_iternext (gen=0x7ffff30626e0, gen@entry=<error reading variable: value has been optimized out>) at ../Objects/genobject.c:283
#26 0x00000000004c96a9 in wrap_next.25272 (self=<optimized out>, args=<optimized out>, wrapped=<optimized out>) at ../Objects/typeobject.c:4683
#27 0x00000000005244dd in PyObject_Call (kw=0x0, arg=(), func=<method-wrapper at remote 0x7ffff3050ad0>) at ../Objects/abstract.c:2529
#28 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffc6b0, func=<method-wrapper at remote 0x7ffff3050ad0>) at ../Python/ceval.c:4238
#29 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc6b0) at ../Python/ceval.c:4043
#30 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffff2fbf5c0, for file /usr/lib/python2.7/contextlib.py, line 17, in __enter__ (self=<GeneratorContextManager(gen=<generator at remote 0x7ffff30626e0>) at remote 0x7ffff304a790>),
throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#31 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218368576, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#32 function_call (func=func@entry=<function at remote 0x7ffff6af2c80>, arg=arg@entry=(<GeneratorContextManager(gen=<generator at remote 0x7ffff30626e0>) at remote 0x7ffff304a790>,), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#33 0x00000000004c2604 in PyObject_Call (kw=0x0, arg=(<GeneratorContextManager(gen=<generator at remote 0x7ffff30626e0>) at remote 0x7ffff304a790>,), func=<function at remote 0x7ffff6af2c80>) at ../Objects/abstract.c:2529
#34 instancemethod_call.8802 (func=<function at remote 0x7ffff6af2c80>, arg=(<GeneratorContextManager(gen=<generator at remote 0x7ffff30626e0>) at remote 0x7ffff304a790>,), kw=0x0) at ../Objects/classobject.c:2602
#35 0x00000000004c6e74 in PyObject_Call (kw=0x0, arg=(), func=<instancemethod at remote 0x7ffff4812aa0>) at ../Objects/abstract.c:2529
#36 PyObject_CallFunctionObjArgs (callable=callable@entry=<instancemethod at remote 0x7ffff4812aa0>) at ../Objects/abstract.c:2760
#37 0x0000000000526020 in PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff305d638, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/postgresql.py, line 53, in delete_database (connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', database='nghtdef_frontend', hostname='**censored ip**'), throwflag=throwflag@entry=0) at ../Python/ceval.c:2555
#38 0x00000000005247ea in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0x7fffffffcd30, func=<function at remote 0x7ffff30617d0>) at ../Python/ceval.c:4106
---Type <return> to continue, or q <return> to quit---[
#39 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcd30) at ../Python/ceval.c:4041
#40 PyEval_EvalFrameEx (
f=f@entry=Frame 0xdac250, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/__init__.py, line 159, in delete (config={u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', database_uris=('postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), connection_args=<ParseResult(database='') at remote 0x7ffff30f3e60>, scheme=u'postgresql', delete_user=<function at remote 0x7ffff3061758>, delete_database=<function at remote 0x7ffff30617d0>, exceptions=[], database_uri='postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', db_args=<ParseResult(database='nghtdef_frontend') at remote 0x7ffff3061848>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#41 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=14336592, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#42 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#43 0x0000000000525cb7 in PyObject_Call (kw={},
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), func=<function at remote 0x7ffff30f3c80>) at ../Objects/abstract.c:2529
#44 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffcf30, func=<function at remote 0x7ffff30f3c80>) at ../Python/ceval.c:4333
#45 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbf960, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 173, in run (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_fronte...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#46 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218367648, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#47 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#48 0x0000000000525cb7 in PyObject_Call (kw={},
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), func=<function at remote 0x7ffff50f4aa0>) at ../Objects/abstract.c:2529
#49 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd130, func=<function at remote 0x7ffff50f4aa0>) at ../Python/ceval.c:4333
#50 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbf790, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 170, in __call__ (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_f...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#51 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218368112, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#52 function_call (func=func@entry=<function at remote 0x7ffff50f4a28>,
arg=arg@entry=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#53 0x00000000004c2604 in PyObject_Call (kw=0x0,
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), func=<function at remote 0x7ffff50f4a28>) at ../Objects/abstract.c:2529
---Type <return> to continue, or q <return> to quit---
#54 instancemethod_call.8802 (func=<function at remote 0x7ffff50f4a28>, func@entry=<instancemethod at remote 0x7ffff4856140>,
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated),
arg@entry=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), kw=kw@entry=0x0) at ../Objects/classobject.c:2602
#55 0x00000000004d26cf in PyObject_Call (kw=0x0,
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), func=<instancemethod at remote 0x7ffff4856140>)
at ../Objects/abstract.c:2529
#56 slot_tp_call.25849 (self=<optimized out>,
args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), kwds=0x0) at ../Objects/typeobject.c:5432
#57 0x0000000000525cb7 in PyObject_Call (kw=0x0,
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'),
func=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>) at ../Objects/abstract.c:2529
#58 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd650,
func=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff30f3c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff2fb9b90>) at ../Python/ceval.c:4333
#59 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff311ada8, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/__init__.py, line 74, in delete_all (config_name='postgresql', prefix='nghtdef', config={u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', databases={'frontend_web': 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'graphite_web': 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'}),
throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#60 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-216945240, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#61 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#62 0x0000000000525cb7 in PyObject_Call (kw={}, arg=('postgresql', 'nghtdef'), func=<function at remote 0x7ffff30f3b90>) at ../Objects/abstract.c:2529
#63 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd850, func=<function at remote 0x7ffff30f3b90>) at ../Python/ceval.c:4333
#64 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbf3f0, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 173, in run (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff30f3b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff30eec10>, args=('postgresql', 'nghtdef'), kwargs={}), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#65 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218369040, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#66 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#67 0x0000000000525cb7 in PyObject_Call (kw={},
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff30f3b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff30eec10>, 'postgresql', 'nghtdef'), func=<function at remote 0x7ffff50f4aa0>) at ../Objects/abstract.c:2529
#68 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffda50, func=<function at remote 0x7ffff50f4aa0>) at ../Python/ceval.c:4333
#69 PyEval_EvalFrameEx (
f=f@entry=Frame 0xf2a6f0, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 276, in _execute (task=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff30f3b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff30eec10>, host='localhost', my_env={'clean_revert': True, 'all_hosts': ['localhost'], 'effective_roles': [], 'command': 'external_database.delete_all'}, args=('postgresql', 'nghtdef'), kwargs={}, jobs=<JobQueue(_finished=False, _queued=[], _running=[], _debug=False, _num_of_jobs=0, _closed=False, _max=1, _comms_queue=None, _completed=[]) at remote 0x7ffff2fb9d10>, queue=None, multiprocessing=None, local_env={'effective_roles': [...], 'all_hosts': [...], 'port': '22', 'clean_revert': True, 'host': 'localhost', 'command': 'external_database.delete_all', 'user': 'root', 'host_string': 'localhost'}),
---Type <return> to continue, or q <return> to quit---
throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#70 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff50bdf30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=8, kws=<optimized out>, kwcount=kwcount@entry=0, defs=defs@entry=0x0,
defcount=defcount@entry=0, closure=0x0) at ../Python/ceval.c:3252
#71 0x0000000000525560 in fast_function (nk=0, na=8, n=<optimized out>, pp_stack=0x7fffffffdc40, func=<function at remote 0x7ffff50f4cf8>) at ../Python/ceval.c:4116
#72 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdc40) at ../Python/ceval.c:4041
#73 PyEval_EvalFrameEx (
f=f@entry=Frame 0xdabd50, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 386, in execute (task=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff30f3b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff30eec10>, args=('postgresql', 'nghtdef'), kwargs={'hosts': [], 'roles': [], 'exclude_hosts': []}, my_env={'clean_revert': True, 'all_hosts': ['localhost'], 'effective_roles': [], 'command': 'external_database.delete_all'}, results={}, is_callable=False, new_kwargs={}, hosts=[...], roles=[...], exclude_hosts=[...], parallel=False, multiprocessing=None, pool_size=1, queue=None, jobs=<JobQueue(_finished=False, _queued=[], _running=[], _debug=False, _num_of_jobs=0, _closed=False, _max=1, _comms_queue=None, _completed=[]) at remote 0x7ffff2fb9d10>, host='localhost'), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#74 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=14335312, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#75 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#76 0x0000000000525cb7 in PyObject_Call (kw={'hosts': [], 'roles': [], 'exclude_hosts': []}, arg=('external_database.delete_all', 'postgresql', 'nghtdef'), func=<function at remote 0x7ffff50f4de8>) at ../Objects/abstract.c:2529
#77 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffde40, func=<function at remote 0x7ffff50f4de8>) at ../Python/ceval.c:4333
#78 PyEval_EvalFrameEx (
f=f@entry=Frame 0xa138b0, for file /usr/local/lib/python2.7/dist-packages/fabric/main.py, line 756, in main (fabfile_locations=None, parser=<OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--eagerly-disconnect': <Option(_long_opts=['--eagerly-disconnect'], help='disconnect from hosts as soon as possible', callback_args=None, callback=None, default=False, nargs=None, choices=None, dest='eagerly_disconnect', container=<...>, _short_opts=['-e'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7ffff533e3b0>, '--port': <Option(_long_opts=['--port'], help='SSH connection port', callback_args=None, callback=None, default='22', nargs=1, choices=None, dest='port', container=<...>, _short_opts=[], action='store', const=None, callback_kwargs=None, type='string', metavar=None) at remote 0x7ffff50cd950>, '--keepalive': <Option(_long_opts=['--keepalive'], help='enables a keepalive every N seconds', callback_args=None, callback=None, default=0, nargs=1, c...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#79 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff7e9eab0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=<optimized out>, kwcount=kwcount@entry=0,
defs=defs@entry=0x7ffff4e81968, defcount=1, closure=0x0) at ../Python/ceval.c:3252
#80 0x0000000000524338 in fast_function (nk=0, na=0, n=<optimized out>, pp_stack=0x7fffffffe030, func=<function at remote 0x7ffff4e89aa0>) at ../Python/ceval.c:4116
#81 call_function (oparg=<optimized out>, pp_stack=0x7fffffffe030) at ../Python/ceval.c:4041
#82 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffff7e713e0, for file /usr/local/bin/fab, line 11, in <module> (), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#83 0x0000000000567d14 in PyEval_EvalCodeEx (closure=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<� at remote 0x7ffff7e71558>,
globals='Return true if the object is a user-defined function.\n\n Function objects provide these attributes:\n __doc__ documentation string\n __name__ name with which this function was defined\n func_code code object containing compiled function bytecode\n func_defaults tuple of any default values for arguments\n func_doc (same as __doc__)\n func_globals global namespace in which this function was defined\n func_name (same as __name__)', co=0x7ffff7edb1b0) at ../Python/ceval.c:3252
#84 PyEval_EvalCode (locals=<� at remote 0x7ffff7e71558>,
globals='Return true if the object is a user-defined function.\n\n Function objects provide these attributes:\n __doc__ documentation string\n __name__ name with which this function was defined\n func_code code object containing compiled function bytecode\n func_defaults tuple of any default values for arguments\n func_doc (same as __doc__)\n func_globals global namespace in which this function was defined\n func_name (same as __name__)', co=0x7ffff7edb1b0) at ../Python/ceval.c:667
#85 run_mod.42576 (mod=mod@entry=0x977f18, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab",
globals=globals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None},
locals=locals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None}, flags=flags@entry=0x7fffffffe1e0, arena=arena@entry=0x9a4a80) at ../Python/pythonrun.c:1370
#86 0x0000000000465bf4 in PyRun_FileExFlags (fp=fp@entry=0x9ef390, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab", start=start@entry=257,
globals=globals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None},
locals=locals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None}, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0) at ../Python/pythonrun.c:1356
#87 0x000000000046612d in PyRun_SimpleFileExFlags (fp=fp@entry=0x9ef390, filename=<optimized out>, filename@entry=0x7fffffffe66b "/usr/local/bin/fab", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0)
at ../Python/pythonrun.c:948
#88 0x0000000000466229 in PyRun_AnyFileExFlags (fp=fp@entry=0x9ef390, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0) at ../Python/pythonrun.c:752
#89 0x0000000000466d92 in Py_Main (argc=<optimized out>, argv=0x7fffffffe398) at ../Modules/main.c:640
#90 0x00007ffff7815f45 in __libc_start_main (main=0x466e50 <main>, argc=9, argv=0x7fffffffe398, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe388) at libc-start.c:287
#91 0x0000000000577c2e in _start ()
On psycopg2==2.7.1 we get the following segmentation fault:
Program received signal SIGSEGV, Segmentation fault.
__strcmp_ssse3 () at ../sysdeps/x86_64/multiarch/../strcmp.S:209
209 ../sysdeps/x86_64/multiarch/../strcmp.S: No such file or directory.
(gdb) bt
#0 __strcmp_ssse3 () at ../sysdeps/x86_64/multiarch/../strcmp.S:209
#1 0x00007ffff6260699 in getrn (lh=lh@entry=0xa9eb60, data=data@entry=0x1030120, rhash=rhash@entry=0x7fffffff9da8) at lhash.c:432
#2 0x00007ffff6260972 in lh_insert (lh=0xa9eb60, data=0x1030120) at lhash.c:189
#3 0x00007ffff61dc8aa in OBJ_NAME_add (name=0x0, type=2, data=0x7ffff2869ec0 "\264\003") at o_names.c:207
#4 0x00007ffff28f1ff7 in SSL_library_init () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/./libssl-cdf7ba29.so.1.0.2k
#5 0x00007ffff2b62e9c in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-9c51d239.so.5.9
#6 0x00007ffff2b505dd in PQconnectPoll () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-9c51d239.so.5.9
#7 0x00007ffff2b50e3e in ?? () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-9c51d239.so.5.9
#8 0x00007ffff2b516ef in PQconnectdb () from /usr/local/lib/python2.7/dist-packages/psycopg2/.libs/libpq-9c51d239.so.5.9
#9 0x00007ffff2d8c241 in _conn_sync_connect (self=0x7ffff30bcc80) at psycopg/connection_int.c:716
#10 conn_connect (self=self@entry=0x7ffff30bcc80, async=<optimized out>) at psycopg/connection_int.c:812
#11 0x00007ffff2d8cf84 in connection_setup (async=<optimized out>, dsn=<optimized out>, self=0x7ffff30bcc80) at psycopg/connection_type.c:1277
#12 connection_init (obj=obj@entry=<psycopg2.extensions.connection at remote 0x7ffff30bcc80>, args=args@entry=('port=5432 host=**censored ip** password=**censored password** dbname=postgres user=admin',), kwds=kwds@entry=0x0)
at psycopg/connection_type.c:1362
#13 0x000000000055f6db in type_call.25495 (type=<optimized out>, type@entry=0x7ffff2fab460 <connectionType>, args=args@entry=('port=5432 host=**censored ip** password=**censored password** dbname=postgres user=admin',), kwds=kwds@entry=0x0)
at ../Objects/typeobject.c:745
#14 0x00000000004c8a7d in PyObject_Call (kw=0x0, arg=('port=5432 host=**censored ip** password=**censored password** dbname=postgres user=admin',), func=<type at remote 0x7ffff2fab460>) at ../Objects/abstract.c:2529
#15 call_function_tail.5417 (callable=callable@entry=<type at remote 0x7ffff2fab460>, args=('port=5432 host=**censored ip** password=**censored password** dbname=postgres user=admin',)) at ../Objects/abstract.c:2561
#16 0x00000000005ab155 in _PyObject_CallFunction_SizeT (callable=<type at remote 0x7ffff2fab460>, format=format@entry=0x7ffff2da3f60 "s") at ../Objects/abstract.c:2605
#17 0x00007ffff2d860b8 in psyco_connect (self=<optimized out>, args=<optimized out>, keywds=<optimized out>) at psycopg/psycopgmodule.c:111
#18 0x000000000052714b in ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffc340, func=<built-in function _connect>) at ../Python/ceval.c:4330
#19 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff302c850, for file /usr/local/lib/python2.7/dist-packages/psycopg2/__init__.py, line 130, in connect (dsn='port=5432 host=**censored ip** password=**censored password** dbname=postgres user=admin', connection_factory=None, cursor_factory=None, kwargs={'port': 5432, 'host': '**censored ip**', 'password': '**censored password**', 'user': 'admin', 'database': 'postgres'}, kwasync={}), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#20 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff2ff5130, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=<optimized out>, kwcount=kwcount@entry=5,
defs=defs@entry=0x7ffff3018888, defcount=3, closure=0x0) at ../Python/ceval.c:3252
#21 0x0000000000524338 in fast_function (nk=5, na=0, n=<optimized out>, pp_stack=0x7fffffffc530, func=<function at remote 0x7ffff300ff50>) at ../Python/ceval.c:4116
#22 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc530) at ../Python/ceval.c:4041
#23 PyEval_EvalFrameEx (
f=f@entry=Frame 0xef3580, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/postgresql.py, line 21, in create_connection (connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', connection_args=<ParseResult(database='') at remote 0x7ffff2fbc8c0>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#24 0x0000000000567b1e in gen_send_ex.isra.0 (exc=0, arg=0x0) at ../Objects/genobject.c:85
#25 gen_iternext (gen=0x7ffff301e730, gen@entry=<error reading variable: value has been optimized out>) at ../Objects/genobject.c:283
#26 0x00000000004c96a9 in wrap_next.25272 (self=<optimized out>, args=<optimized out>, wrapped=<optimized out>) at ../Objects/typeobject.c:4683
#27 0x00000000005244dd in PyObject_Call (kw=0x0, arg=(), func=<method-wrapper at remote 0x7ffff300ed90>) at ../Objects/abstract.c:2529
#28 do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fffffffc6b0, func=<method-wrapper at remote 0x7ffff300ed90>) at ../Python/ceval.c:4238
#29 call_function (oparg=<optimized out>, pp_stack=0x7fffffffc6b0) at ../Python/ceval.c:4043
#30 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffff2fbb5c0, for file /usr/lib/python2.7/contextlib.py, line 17, in __enter__ (self=<GeneratorContextManager(gen=<generator at remote 0x7ffff301e730>) at remote 0x7ffff2ffaa50>),
throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#31 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218384960, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#32 function_call (func=func@entry=<function at remote 0x7ffff6af2c80>, arg=arg@entry=(<GeneratorContextManager(gen=<generator at remote 0x7ffff301e730>) at remote 0x7ffff2ffaa50>,), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#33 0x00000000004c2604 in PyObject_Call (kw=0x0, arg=(<GeneratorContextManager(gen=<generator at remote 0x7ffff301e730>) at remote 0x7ffff2ffaa50>,), func=<function at remote 0x7ffff6af2c80>) at ../Objects/abstract.c:2529
#34 instancemethod_call.8802 (func=<function at remote 0x7ffff6af2c80>, arg=(<GeneratorContextManager(gen=<generator at remote 0x7ffff301e730>) at remote 0x7ffff2ffaa50>,), kw=0x0) at ../Objects/classobject.c:2602
#35 0x00000000004c6e74 in PyObject_Call (kw=0x0, arg=(), func=<instancemethod at remote 0x7ffff4810aa0>) at ../Objects/abstract.c:2529
#36 PyObject_CallFunctionObjArgs (callable=callable@entry=<instancemethod at remote 0x7ffff4810aa0>) at ../Objects/abstract.c:2760
#37 0x0000000000526020 in PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff300b638, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/postgresql.py, line 53, in delete_database (connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', database='nghtdef_frontend', hostname='**censored ip**'), throwflag=throwflag@entry=0) at ../Python/ceval.c:2555
#38 0x00000000005247ea in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0x7fffffffcd30, func=<function at remote 0x7ffff2fbc758>) at ../Python/ceval.c:4106
---Type <return> to continue, or q <return> to quit---
#39 call_function (oparg=<optimized out>, pp_stack=0x7fffffffcd30) at ../Python/ceval.c:4041
#40 PyEval_EvalFrameEx (
f=f@entry=Frame 0xef27f0, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/__init__.py, line 159, in delete (config={u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', database_uris=('postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), connection_args=<ParseResult(database='') at remote 0x7ffff3070e60>, scheme=u'postgresql', delete_user=<function at remote 0x7ffff2fbc6e0>, delete_database=<function at remote 0x7ffff2fbc758>, exceptions=[], database_uri='postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', db_args=<ParseResult(database='nghtdef_frontend') at remote 0x7ffff2fbc7d0>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#41 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=15673328, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#42 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#43 0x0000000000525cb7 in PyObject_Call (kw={},
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), func=<function at remote 0x7ffff3070c80>) at ../Objects/abstract.c:2529
#44 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffcf30, func=<function at remote 0x7ffff3070c80>) at ../Python/ceval.c:4333
#45 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbb960, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 173, in run (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_fronte...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#46 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218384032, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#47 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#48 0x0000000000525cb7 in PyObject_Call (kw={},
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), func=<function at remote 0x7ffff50f4aa0>) at ../Objects/abstract.c:2529
#49 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd130, func=<function at remote 0x7ffff50f4aa0>) at ../Python/ceval.c:4333
#50 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbb790, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 170, in __call__ (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_f...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#51 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218384496, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#52 function_call (func=func@entry=<function at remote 0x7ffff50f4a28>,
arg=arg@entry=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), kw=kw@entry=0x0) at ../Objects/funcobject.c:526
#53 0x00000000004c2604 in PyObject_Call (kw=0x0,
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated), func=<function at remote 0x7ffff50f4a28>) at ../Objects/abstract.c:2529
---Type <return> to continue, or q <return> to quit---
#54 instancemethod_call.8802 (func=<function at remote 0x7ffff50f4a28>, func@entry=<instancemethod at remote 0x7ffff4855140>,
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>, {u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**...(truncated),
arg@entry=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), kw=kw@entry=0x0) at ../Objects/classobject.c:2602
#55 0x00000000004d26cf in PyObject_Call (kw=0x0,
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), func=<instancemethod at remote 0x7ffff4855140>)
at ../Objects/abstract.c:2529
#56 slot_tp_call.25849 (self=<optimized out>,
args=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'), kwds=0x0) at ../Objects/typeobject.c:5432
#57 0x0000000000525cb7 in PyObject_Call (kw=0x0,
arg=({u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, 'postgresql://admin:**censored password**@**censored ip**:5432/', 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'),
func=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>) at ../Objects/abstract.c:2529
#58 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd650,
func=<WrappedCallableTask(__module__='fabfile.external_database', name='delete', is_default=False, wrapped=<function at remote 0x7ffff3070c80>, __name__='delete', __doc__='Delete one or more databases on an external database server.\n\n Provide a connection_uri which can delete users and databases.\n And one or more database_uri for the databases and users to be deleted.\n\n The uri should be formatted as follows:\n <scheme>://<username>:<password>@<hostname>[:<port>]/<database>\n With scheme one of "postgresql", "mysql" or "mssql".\n In the case of the connection_uri, there won\'t be a /<database> part.\n ') at remote 0x7ffff3034e90>) at ../Python/ceval.c:4333
#59 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff3118da8, for file /home/ubuntu/workspace/external-database-create/awingu/clouddesktop-reposerver/fabfile/external_database/__init__.py, line 74, in delete_all (config_name='postgresql', prefix='nghtdef', config={u'username': u'admin', u'conn_db': u'', u'protocol': u'postgresql', u'host': u'**censored ip**', u'import_path': u'postgresql', u'password': u'**censored password**', u'port': 5432}, connection_uri='postgresql://admin:**censored password**@**censored ip**:5432/', databases={'frontend_web': 'postgresql://nghtdef_frontend:nghtdef_frontend@**censored ip**:5432/nghtdef_frontend', 'graphite_web': 'postgresql://nghtdef_graphite:nghtdef_graphite@**censored ip**:5432/nghtdef_graphite'}),
throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#60 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-216953432, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#61 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#62 0x0000000000525cb7 in PyObject_Call (kw={}, arg=('postgresql', 'nghtdef'), func=<function at remote 0x7ffff3070b90>) at ../Objects/abstract.c:2529
#63 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffd850, func=<function at remote 0x7ffff3070b90>) at ../Python/ceval.c:4333
#64 PyEval_EvalFrameEx (
f=f@entry=Frame 0x7ffff2fbb3f0, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 173, in run (self=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff3070b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff3034e10>, args=('postgresql', 'nghtdef'), kwargs={}), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#65 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=-218385424, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#66 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#67 0x0000000000525cb7 in PyObject_Call (kw={},
arg=(<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff3070b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff3034e10>, 'postgresql', 'nghtdef'), func=<function at remote 0x7ffff50f4aa0>) at ../Objects/abstract.c:2529
#68 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffda50, func=<function at remote 0x7ffff50f4aa0>) at ../Python/ceval.c:4333
#69 PyEval_EvalFrameEx (
f=f@entry=Frame 0xca61d0, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 276, in _execute (task=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff3070b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff3034e10>, host='localhost', my_env={'clean_revert': True, 'all_hosts': ['localhost'], 'effective_roles': [], 'command': 'external_database.delete_all'}, args=('postgresql', 'nghtdef'), kwargs={}, jobs=<JobQueue(_finished=False, _queued=[], _running=[], _debug=False, _num_of_jobs=0, _closed=False, _max=1, _comms_queue=None, _completed=[]) at remote 0x7ffff3034f50>, queue=None, multiprocessing=None, local_env={'effective_roles': [...], 'all_hosts': [...], 'port': '22', 'clean_revert': True, 'host': 'localhost', 'command': 'external_database.delete_all', 'user': 'root', 'host_string': 'localhost'}),
---Type <return> to continue, or q <return> to quit---
throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#70 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff50bdf30, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=8, kws=<optimized out>, kwcount=kwcount@entry=0, defs=defs@entry=0x0,
defcount=defcount@entry=0, closure=0x0) at ../Python/ceval.c:3252
#71 0x0000000000525560 in fast_function (nk=0, na=8, n=<optimized out>, pp_stack=0x7fffffffdc40, func=<function at remote 0x7ffff50f4cf8>) at ../Python/ceval.c:4116
#72 call_function (oparg=<optimized out>, pp_stack=0x7fffffffdc40) at ../Python/ceval.c:4041
#73 PyEval_EvalFrameEx (
f=f@entry=Frame 0xf97530, for file /usr/local/lib/python2.7/dist-packages/fabric/tasks.py, line 386, in execute (task=<WrappedCallableTask(__module__='fabfile.external_database', name='delete_all', is_default=False, wrapped=<function at remote 0x7ffff3070b90>, __name__='delete_all', __doc__=None) at remote 0x7ffff3034e10>, args=('postgresql', 'nghtdef'), kwargs={'hosts': [], 'roles': [], 'exclude_hosts': []}, my_env={'clean_revert': True, 'all_hosts': ['localhost'], 'effective_roles': [], 'command': 'external_database.delete_all'}, results={}, is_callable=False, new_kwargs={}, hosts=[...], roles=[...], exclude_hosts=[...], parallel=False, multiprocessing=None, pool_size=1, queue=None, jobs=<JobQueue(_finished=False, _queued=[], _running=[], _debug=False, _num_of_jobs=0, _closed=False, _max=1, _comms_queue=None, _completed=[]) at remote 0x7ffff3034f50>, host='localhost'), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#74 0x0000000000568b3a in PyEval_EvalCodeEx (closure=<optimized out>, defcount=<optimized out>, defs=0x0, kwcount=<optimized out>, kws=<optimized out>, argcount=16348464, args=<optimized out>, locals=0x0, globals=<optimized out>,
co=<optimized out>) at ../Python/ceval.c:3252
#75 function_call (func=<optimized out>, arg=<optimized out>, kw=<optimized out>) at ../Objects/funcobject.c:526
#76 0x0000000000525cb7 in PyObject_Call (kw={'hosts': [], 'roles': [], 'exclude_hosts': []}, arg=('external_database.delete_all', 'postgresql', 'nghtdef'), func=<function at remote 0x7ffff50f4de8>) at ../Objects/abstract.c:2529
#77 ext_do_call (nk=<optimized out>, na=<optimized out>, flags=<optimized out>, pp_stack=0x7fffffffde40, func=<function at remote 0x7ffff50f4de8>) at ../Python/ceval.c:4333
#78 PyEval_EvalFrameEx (
f=f@entry=Frame 0xa138b0, for file /usr/local/lib/python2.7/dist-packages/fabric/main.py, line 756, in main (fabfile_locations=None, parser=<OptionParser(process_default_values=True, allow_interspersed_args=True, _long_opt={'--eagerly-disconnect': <Option(_long_opts=['--eagerly-disconnect'], help='disconnect from hosts as soon as possible', callback_args=None, callback=None, default=False, nargs=None, choices=None, dest='eagerly_disconnect', container=<...>, _short_opts=['-e'], action='store_true', const=None, callback_kwargs=None, type=None, metavar=None) at remote 0x7ffff533e3b0>, '--port': <Option(_long_opts=['--port'], help='SSH connection port', callback_args=None, callback=None, default='22', nargs=1, choices=None, dest='port', container=<...>, _short_opts=[], action='store', const=None, callback_kwargs=None, type='string', metavar=None) at remote 0x7ffff50cd950>, '--keepalive': <Option(_long_opts=['--keepalive'], help='enables a keepalive every N seconds', callback_args=None, callback=None, default=0, nargs=1, c...(truncated), throwflag=throwflag@entry=0) at ../Python/ceval.c:2705
#79 0x0000000000555551 in PyEval_EvalCodeEx (co=0x7ffff7e9eab0, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=0, kws=<optimized out>, kwcount=kwcount@entry=0,
defs=defs@entry=0x7ffff4e81968, defcount=1, closure=0x0) at ../Python/ceval.c:3252
#80 0x0000000000524338 in fast_function (nk=0, na=0, n=<optimized out>, pp_stack=0x7fffffffe030, func=<function at remote 0x7ffff4e89aa0>) at ../Python/ceval.c:4116
#81 call_function (oparg=<optimized out>, pp_stack=0x7fffffffe030) at ../Python/ceval.c:4041
#82 PyEval_EvalFrameEx (f=f@entry=Frame 0x7ffff7e713e0, for file /usr/local/bin/fab, line 11, in <module> (), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#83 0x0000000000567d14 in PyEval_EvalCodeEx (closure=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=<� at remote 0x7ffff7e71558>,
globals='Return true if the object is a user-defined function.\n\n Function objects provide these attributes:\n __doc__ documentation string\n __name__ name with which this function was defined\n func_code code object containing compiled function bytecode\n func_defaults tuple of any default values for arguments\n func_doc (same as __doc__)\n func_globals global namespace in which this function was defined\n func_name (same as __name__)', co=0x7ffff7edb1b0) at ../Python/ceval.c:3252
#84 PyEval_EvalCode (locals=<� at remote 0x7ffff7e71558>,
globals='Return true if the object is a user-defined function.\n\n Function objects provide these attributes:\n __doc__ documentation string\n __name__ name with which this function was defined\n func_code code object containing compiled function bytecode\n func_defaults tuple of any default values for arguments\n func_doc (same as __doc__)\n func_globals global namespace in which this function was defined\n func_name (same as __name__)', co=0x7ffff7edb1b0) at ../Python/ceval.c:667
#85 run_mod.42576 (mod=mod@entry=0x977f18, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab",
globals=globals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None},
locals=locals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None}, flags=flags@entry=0x7fffffffe1e0, arena=arena@entry=0x9a4a80) at ../Python/pythonrun.c:1370
#86 0x0000000000465bf4 in PyRun_FileExFlags (fp=fp@entry=0x9ef390, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab", start=start@entry=257,
globals=globals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None},
locals=locals@entry={'__builtins__': <module at remote 0x7ffff7fa1b08>, '__file__': '/usr/local/bin/fab', '__package__': None, 'sys': <module at remote 0x7ffff7fa1bb0>, 're': <module at remote 0x7ffff7ed32b8>, '__name__': '__main__', 'main': <function at remote 0x7ffff4e89aa0>, '__doc__': None}, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0) at ../Python/pythonrun.c:1356
#87 0x000000000046612d in PyRun_SimpleFileExFlags (fp=fp@entry=0x9ef390, filename=<optimized out>, filename@entry=0x7fffffffe66b "/usr/local/bin/fab", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0)
at ../Python/pythonrun.c:948
#88 0x0000000000466229 in PyRun_AnyFileExFlags (fp=fp@entry=0x9ef390, filename=filename@entry=0x7fffffffe66b "/usr/local/bin/fab", closeit=closeit@entry=1, flags=flags@entry=0x7fffffffe1e0) at ../Python/pythonrun.c:752
#89 0x0000000000466d92 in Py_Main (argc=<optimized out>, argv=0x7fffffffe398) at ../Modules/main.c:640
#90 0x00007ffff7815f45 in __libc_start_main (main=0x466e50 <main>, argc=9, argv=0x7fffffffe398, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe388) at libc-start.c:287
#91 0x0000000000577c2e in _start ()
I think I know what it is, and it is not pretty :(
#4 0x00007ffff28f1ff7 in SSL_library_init () from
/usr/local/lib/python2.7/dist-packages/psycopg2/.libs/./libssl-cdf7ba29.so.1.0.2k
you are using a wheel package, which is self-contained and is bundled with its own version of libssl. Python uses the system version of the library. The two libraries together probably clash on some global object, or step onto each other's private structure.
I think the problem cannot be solved, which is sad: the wheel package is probably not compatible with the ssl module, or whatever else binds to openssl. The workaround is to install psycopg from source, e.g. using pip install --no-binary psycopg2. I'm leaving this issue open because we should do some of the following:
- document the issue
- detect the problem on connect(), i.e. that
sslis loaded so we can't use crypt. Or something else? This wouldn't avoid crashing if psycopg is loaded beforesslthough.
Thorny problem.
Improve docs by 2.7.2
Wow. This one burned me for a couple of hours. All I was getting was "segmentation fault" on ubuntu 16.04 with a tornado + aiopg app. I have to investigate the core to get here. Rolling back to psycopg2==2.6.2 seems to have resolved things for me.
@oliverseal Yeah, that's very unfortunate and unfortunately I don't see a way to solve it. You can still use 2.7 forcing source installation.
I've added a warning in the docs about the problem, but if you had to dig deeper it may mean that warning is not enough. Please feel free to suggest how we can improve the docs further to help people not waste too much time.
Would it be possible to not bundle libssl in the wheel and use the one that comes with python instead?
@rubendv I don't think so because it's the libpq to bind to the libssl, not psycopg directly.
Maybe there could be a way to exclude the libssl from the wheel generation; then we should check the binary compatibility libssl offers. It would be actually interesting to take a look into that.
Why keep the wheel, in the meantime, given how broken it is? Building from source should work for far more environments than the wheel, or at least, error out if the appropriate libraries / headers are available. Presently, naïvely attempting to install psycopg2 would appear to just be broken for many people (it appears to be broken on Ubuntu, for example), and for those of us deploying in automated fashions, adding --no-binary would seem to imply the huge toll of building everything from source.
@thanatos I have maintained this library for more 10 years and I can tell you that people incapable of reading the first line of the manual, of installing the -dev packages, opening bugs because of the mysterious writing "I can't find pg_config: please add it to the path" are an endless army.
The wheel is presently only broken for people also using libssl, which are a minority. People who have automated deployment by just sticking psycopg2 in requirements.txt have done it wrong: if you depend on psycopg2 you are guaranteed something will break if the behaviour of the library changes. A more conservative approach would be to depend on psycopg2>=2.6,<2.7 through which you would have only received bugfixes, not behavioural changes, and you wouldn't have received the wheel package. Now that you have tested that psycopg 2.7 works for your program compiled from source please use psycopg2>=2.7,<2.8 --no-binary :all: in your requirements, and yes, rebuild everything: as you have automated it it's just pressing a button.
For every person complaining about this segfault I have 100 OSX dummies who stopped complaining. It is a net win for me.
Issue re-opened, because documenting a workaround doesn't effectively close it.
The wheel is presently only broken for people also using libssl, which are a minority.
My understanding is that the standard library's ssl module links against the system libssl. Now, some people might be getting lucky, in that they never import that module, but I just don't think it is unreasonable to ask that libraries such as psycopg2 to work regardless of whether or not I import certain libraries from the standard library, within reason. (Perhaps some dependency of a dependency in my stack is importing ssl, but how much control I have over that, I don't know, and I don't believe I have either a good way of detecting who that is or even doing anything about it.)
would be to depend on
psycopg2>=2.6,<2.7through which you would have only received bugfixes, not behavioural changes
My options here are also fairly limited: we actually do attempt to pin dependencies s.t. we don't pull in behavioral changes inadvertently. However, we have a large number of dependencies, and our time is severely limited; we don't always have time to chase down libraries to see if they use some custom, home-grown versioning scheme, and what that is. For the most part, we don't bother, as it is usually not specified anywhere and thus a fool's errand; my life would be a lot easier if folks would just use semver (and if pip would get better support for it, such as what cargo or npm have). In particular, I don't see a good reason for psycopg2 to not follow it, and it also doesn't appear to be explicitly stated anywhere, though perhaps one could argue that it could be inferred from the changelog. (The point of movements such as semver, however, is to streamline stuff and reduce the amount of inference and guesswork that I need to do.)
psycopg2>=2.7,<2.8 --no-binary :all:
I wasn't aware that this was even valid syntax for a requirements file. However, it unfortunately seems to apply to the entire set, not the single requirement (and thus doesn't work around the issue of subsequently needing to rebuild everything). (If I make a requirements file with that and a single other requirement, pip outputs that it is skipping wheels for both, and all of their dependencies.)
This also ignores that things being pulled in through other dependencies are going to need to push their dependencies through setup.py, and I'm not sure if it provides a similar mechanism.
Issue re-opened
Thank you.
For every person complaining about this segfault I have 100 OSX dummies who stopped complaining. It is a net win for me.
I'm not against bundling the postgres library; that seems fairly safe to do; it's the bundled libssl, since it so heavily conflicts with the standard library's. libssl is essentially guaranteed to exist (since the stdlib links against it) so it would seem to me that leaving that library alone out would solve both the segfaults, and folks who can't understand how to build from source from bugging you, would it not?
Of course I'm not happy that psycopg segfaults in any condition, let alone in conjunction with the use of a stdlib module. It would be great if we could exclude libssl from the wheel package; unfortunately this is not possible because libssl doesn't have a stable ABI. It is true that it pretty much exists everywhere; the problem is that we don't know what shape it has.
It would also be great if the workflow could be:
$ pip install psycopg2
... ponder ponder ...
It appears you don't have build tool. Please use "pip install --binary psycopg2" and hope
for the best, alternatively please install the build tools
$ pip install --binary psycopg2
... ponder ponder ...
wheel installed.
however there is no "prefer source" or "prefer binary" option in pip, AFAICS. Maybe it's worth asking them?
As per the issue that --no-binary applies to the whole requirements.txt files, again, it seems a pip shortcoming, not ours.
As per the semantic versioning: we do it. The 2 is there to stay in psycopg2: there will be a psycopg3 to change things heavily. We increase the second number when we add features or change some behaviour in some subtle way. We increase the third number when we fix bug. So yeah, we do semver and you can use the >=2.X,<2.(X+1) pattern on psycopg. If others don't it's not exactly our problem, no?
So to recap, the problem with psycopg wheels is caused by a combination of
- libssl doesn't version its ABI
- pip doesn't allow a package to specify "source preferred"
- pip
--no-binary :all:is too aggressive
on the first being fixed I don't hold my breath. Point three could be just a bug, the second would be nice, but I don't have the energy to lobby the pip guys.
If you can help with some of the above, be my guest. If not, please don't insist that we drop the wheel support because some random guy in pypi rolls three dice to choose their version numbers and you don't feel like adding version guards in your req's file. I'm happy to work on the shortcomings of the libraries I maintain and to give alternatives to work around problems, but I have limited agency upstream (the libraries we use) and downstream (the users we have).
Looked better into all the packages going no-binary in requirements.txt. I thought :all: referred to the entry it appears into only and meant no version but no, it doesn't. Docs are here.
The right solution is to use --no-binary psycopg2. For example this file:
$ cat requirements.txt
psycopg2>=2.7,<2.8 --no-binary psycopg2
lxml
Installs:
$ pip install -r requirements.txt --no-cache
Collecting psycopg2<2.8,>=2.7 (from -r requirements.txt (line 1))
Downloading psycopg2-2.7.3.2.tar.gz (425kB)
Collecting lxml (from -r requirements.txt (line 2))
Downloading lxml-4.1.0-cp27-cp27mu-manylinux1_x86_64.whl (5.6MB)
Installing collected packages: psycopg2, lxml
Running setup.py install for psycopg2 ... done
Successfully installed lxml-4.1.0 psycopg2-2.7.3.2
where lxml is installed from wheel and psycopg from source.
I will update the documentation to suggest this workaround.
I would like to work to solve this problem, but I can't actually reproduce it.
Tried importing libssl both by psycopg and by urllib/requests with a script like the following, which in my understanding should have triggered the segfault
from __future__ import print_function
import requests
s = requests.get('https://github.com')
print("from requests:", len(s.content))
try:
from urllib2 import urlopen
except ImportError:
from urllib.request import urlopen
f = urlopen('https://github.com')
print("from urllib:", len(f.read()))
import psycopg2
cnn = psycopg2.connect('host=localhost sslmode=require')
cur = cnn.cursor()
cur.execute("select 1")
print("from psycopg:", cur.fetchone())checking in /proc/PID/map_files I can see both the libraries are actually imported:
$ ls -l /proc/12994/map_files/ | grep libcrypto
lr-------- 1 piro piro 64 Nov 16 23:52 7fea9fb7f000-7fea9fda5000 -> /home/piro/dev/psycopg2/env3/lib/python3.5/site-packages/psycopg2/.libs/libcrypto-34813f62.so.1.0.2l
[...]
lr-------- 1 piro piro 64 Nov 16 23:52 7feab07a1000-7feab09bb000 -> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
however the program completes without segfaulting.
Tested on Python 2.7.12 and 3.5.2 on Ubuntu 16.04.3 64 bit.
Any suggestion about how to trigger the segfault reliably?
@rubendv would you be able to provide a standalone test to reproduce the issue? I've tried in an ubuntu 14:04 docker image containing:
ii libssl1.0.0:amd64 1.0.1f-1ubuntu2.23 amd64
ii python 2.7.5-5ubuntu3 amd64
I couldn't reproduce the segfault, tested with psycopg 2.7.1 (the one you tested with) and 2.7.3.2 (the current one).
Your test depends on a module whose behaviour is not clear (just importing fabric.api doesn't seem to import _ssl) and #603 seems suggesting that triggering the bug requires a specific order.
A more self-contained test in a dockerfile would be welcome to fix the problem, thank you.
Whilst I can't easily reproduce it, I have experienced the same issue using requests==2.18.4 and psycopg2==2.7.3.1
In my cases, it seems to occur more frequently (and even then, not actually frequently) when requests.ConnectionError is raised (and subsequently caught & handled, such that the program continues)
Below is the debug information I have from the OSX console, which is ... possibly useless? I certainly wouldn't be able to diagnose anything from it.
Given its up to thread 7 (or 8 in some cases) I assume I was running a Django project under gunicorn at the time. I'm fairly certain I had similar crashes using the Django runserver but the data for that doesn't have any relevant information whatsoever.
Path: /Users/USER/*/python3.5
Identifier: python3.5
Code Type: X86-64 (Native)
Parent Process: python3.5 [2896]
OS Version: Mac OS X 10.11.6 (15G1004)
Crashed Thread: 7
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: EXC_I386_GPFLT
Thread 7 Crashed:
0 libssl.1.0.0.dylib 0x0000000102908fae freelist_extract + 89
1 libssl.1.0.0.dylib 0x0000000102908f01 ssl3_setup_read_buffer + 125
2 libssl.1.0.0.dylib 0x000000010290910c ssl3_setup_buffers + 14
3 libssl.1.0.0.dylib 0x000000010290ae40 ssl23_connect + 363
4 libpq.5.8.dylib 0x00000001028e5400 pgtls_open_client + 2186
5 libpq.5.8.dylib 0x00000001028d275d PQconnectPoll + 1360
6 libpq.5.8.dylib 0x00000001028d0eee connectDBComplete + 106
7 libpq.5.8.dylib 0x00000001028d1042 PQconnectdb + 36
8 _psycopg.cpython-35m-darwin.so 0x00000001028973b6 conn_connect + 166
9 _psycopg.cpython-35m-darwin.so 0x0000000102898780 connection_init + 304
10 python 0x0000000100b04b29 type_call + 297
11 python 0x0000000100aad181 PyObject_Call + 97
12 python 0x0000000100aad466 _PyObject_CallFunction_SizeT + 246
13 _psycopg.cpython-35m-darwin.so 0x0000000102892c54 psyco_connect + 180
[...]
165 libsystem_pthread.dylib 0x00007fff8f38e99d _pthread_body + 131
166 libsystem_pthread.dylib 0x00007fff8f38e91a _pthread_start + 168
167 libsystem_pthread.dylib 0x00007fff8f38c351 thread_start + 13
And now what I assume is some assembly:
Thread 7 crashed with X86 Thread State (64-bit):
rax: 0x00007fe706160c10 rbx: 0x00007fe706160380 rcx: 0x0000000000000266 rdx: 0x000000010292ed31
rdi: 0x0000000000000009 rsi: 0x000000000000000c rbp: 0x00007000023a1d40 rsp: 0x00007000023a1d20
r8: 0x0000000000005600 r9: 0x0000000000000000 r10: 0x000000005b32b633 r11: 0x00007fe702a00000
r12: 0x0000000000000000 r13: 0x00007fe702aa3770 r14: 0xe500030317818c00 r15: 0x0000000000004548
rip: 0x0000000102908fae rfl: 0x0000000000010286 cr2: 0x00002df652a05000
I appreciate that more anecdata probably doesn't offer up any new information or help, but just in case :)
Observation:
SSL_library_initis not reentrant, and all the reports seem suggesting multithread environment (gunicorn, fabfile).libpqcallsSSL_library_initat every connection (why, even?...)libpqcallsSSL_library_initonly in theCONNECTION_SSL_STARTUPstep- If this is a concurrency problem (as it smells), connecting with the GIL should be enough to avoid problems, but it's probably a bad hit for performance.
- If we call
PQconnectdbon connection there is no way to isolateCONNECTION_SSL_STARTUP. - But if we use
PQconnectPolland do the wait ourselves we may isolateCONNECTION_SSL_STARTUPand reacquire the lock just there - We use
PQconnectdbin normal psycopg mode, we usePQconnectPollin green and async mode - We could ditch
PQconnectdbaltogether and roll our own with our ownPQconnectPoll+ wait, protecting the SSL step.
Starting from psycopg 2.7.4, a new package will be introduced: psycopg2-binary. The psycopg2 wheel package will be still available but will importing it will raise an explanatory warning.
Starting from psycopg 2.8, only psycopg2-binary will be available as wheel package and 'pip install psycopg2' will require build tools on the client. In case of missing build prerequisites, setup.py will also inform about the availability of the psycopg2-binary package.
@dvarrazzo Did you try to use static linking for build libpq instead of dynamic linking? Maybe it would help to solve this issue.
@Cykooz I found problem building the libs statically. And I haven't managed to reproduce the issue reliably so I figured out "that's a solution I'm failing to put together and if I even can I'll fail to figure out if it worked for real". So didn't insist much. If you want to test something yourself let me know.
Hi contributors,
This issue triggers me and I also spend some time trying to figure out how to solve it.
In fact, SSL_library_init() is just a bunch of OBJ_NAME_add().
The original stack trace provided by @rubendv shows that it segfaults inside OBJ_NAME_add(), wich is used only at initialization.
The change [1] shows that a mutex had to be placed to avoid concurrent issues when accessing the OBJ_NAME table because objects can be added concurrently. And also isn't expected to add algorithms again and again as libpg does. However [1] don't seems to address the initialization issue completly. My guess is that a synchronization mechanism can't be added fully because chyphers can be aquired at any time so it would become a performace issue.
In fact even if the libssl isn't versionated it should be pretty stable. They even has a tool [2] to deal with it. So before give up and assume that isn't safe to have different versions at the same time, as it always worked as it was until this bug.
My sugestion is to change OBJ_NAME_add() to OBJ_NAME_get() first, my guess is that will fix it. Would be a minor change with no impact.
As nobody know how to reproduce this bug, I don't ever bothered to patch it. Anyway it is just point OBJ_NAME_get() return value to *data of OBJ_NAME_add(const char *name, int type, const char *data).
[1] openssl/openssl#3525
[2] https://wiki.openssl.org/index.php/Binary_Compatibility
Hi @pslacerda,
thank you for looking into this. My attempts to reproduce the bug were misguided because I thought it was only a matter of using the two libraries together, whereas it should be tested in multithread environment.
As far as I can see, the wheel distribution has several other problems, see the list of bugs attributed to it. As such I'm reluctant to try and make the thing work just to discover it will fail under even more subtle but reasonable conditions.
@dvarrazzo, I agree. A wheel package then is a lot safer.
However maybe you can put it in the next major version. As already discussed, some precarious code around there will break with this change (swap the contents from wheel to source) because there are people unable to compile source packages and don't stick to determined versions.
Semantic versioning don't bother with this, it only requires that the major version must be incremented when incompatible changes are introduced, but not forbids increment it otherwise.
Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when major version is incremented.
So is a lot safer to just increment the major version as you both fix this nasty bug class and also avoid break existing noob code. Keep that ugly warning to ensure that who cares will port to the next version If you have an schedule already set up than do it on the next minor version, otherwise right now.
auditwheel goes to great lengths to make sure that libraries used by different packages don't conflict. In particular, it should be impossible for the stdlib ssl module to be affected by a Linux psycopg2 wheel, or vice-versa, unless you do wacky stuff like force a library to be loaded with RTLD_GLOBAL. The same is true (for different reasons) with MacOS wheels if they're built correctly (in particular avoiding using the "flat namespace" for SSL symbols). Windows is trickier, but it doesn't look like any of the folks hitting this above are on Windows.
My first suspicion would be that the binary wheels thing is a red herring, and it's really a plain old threading bug (that happens to be more or less likely to hit based on random variations in the environment, like whether you're using the system version of ssl or the one that ships in the wheels).
@njsmith there is enough evidence to point to that direction.
1: While I haven't been able to reproduce the issue consistently, users who do have started complaining since they started using wheels (accidentally, in the 2.7 released) and stopped complaining since they migrated away from wheels.
2: if you check the OP traceback, it points straight to the area where the libpq (postgres client library) initialize openssl to connect. Libpq manages the use of the non-reentrant openssl init function by itself (using a lock for serialization) and failures only happen in conjunction with the rest of python doing concurrent https requests.
there are several other issues which seem related to wheels only (on Linux and OSX) and this type of problem just started cropping up when we introduced wheels. You are very welcome to help by taking a look and check if we are building the wheel packages or related libraries the wrong way, but the fact that these problems are related to the use of the wheel artifact is pretty undeniable.
Well, what I can say for sure is that if you're having symbol collisions between the stdlib's openssl and your openssl, then that is definitely something we would consider a bug, and if you can demonstrate it then we'll try to fix it. But this still doesn't really look like that kind of issue: generally if you mix symbols from different libraries then it crashes immediately, for everyone. I can't think of a mechanism that would allow the two openssl libraries to interfere with each other in the wheel builds. The two libraries are loaded separately, into different symbol namespaces, they get separate memory regions, separate copies of all global variables, etc. And there are many other packages wheels that aren't seeing any problems like this, including packages shipping openssl (like cryptography), and packages shipping incompatible versions of other libraries (e.g. numpy and scipy sometimes ship different versions of libopenblas, without trouble). So obviously something is going on, but it seems unlikely that it's a simple matter of symbol collisions.
Here's another possibility that I thought of after sleeping on it: when people don't use wheels, they'll generally end up with the stdlib ssl and psycopg2 both using the same (system) openssl. With wheels, you end up with two separate copies of openssl. What if it's not so much that having two openssls causes problems, as that having a single shared openssl has some kind of protective effect, and covers up a bug?
More specifically, here's a possibility: in older versions of openssl – including the ones that appear in those crash dumps above – libssl is not thread-safe by default, i.e. using it from two different threads will sometimes cause weird crashes. You can make it thread safe by setting up special "locking callbacks". The Python stdlib ssl module does set up these callbacks. So, if you do import ssl, then afterwards, all code that's in the same process and using the same copy of libssl can safely call libssl functions from multiple threads. So if psycopg2 doesn't set up these callbacks properly, then so long as it's using the system libssl it won't matter, because it can piggyback on the stdlib ssl module. But, when you have a psycopg2 wheel, it has its own copy of libssl, and now psycopg2 has to set up the locking callbacks correctly or else it will see weird crashes when using multiple threads. Obviously this is a shot in the dark, but it does it sound plausible?
This idea is partially inspired by Paul Kehrer's IRC explanation of how cryptography handles these callbacks:
cryptography does the following: on import it tries to
__import__("_ssl"). Once it does that it checks to see if locking callbacks are registered. If they are then cryptography leaves them alone. If they do not exist then we register our own, which are derived from the CPython locking callbacks + modifications made by arigato so that they don't depend on CPython's internals (and thus work on pypy)
finally, if it's OpenSSL 1.1.0+ we don't register locking callbacks at all because OpenSSL handles that for us
so if we're compiled against the same OpenSSL that Python is compiled against we basically never register our own locking callbacks and instead rely on Python to do that for us. In an embedded scenario we may not register callbacks as long as the process embedding us has registered their own before we're imported
(If this is the issue, then the easiest fix is probably to upgrade the wheels to openssl 1.1.0, rather than messing around with trying to set up the locking callbacks correctly. And using the latest openssl version is probably a good idea anyway...)
Hi @njsmith
psycopg configures the libpq to use the Python callbacks as per commit a59704c. So it seems it plays already well in this regard.
I haven't tested with libcrypto 1.1: compiling 1.0 on the Centos "Nevrotic Neanderthal" Release needed by the wheels build infrastructure was already enough of a sink of time for me. By then libpq support for libcrypto 1.1 was still in its infancy, and Ubuntu 16.04 didn't support it either, which made me sound the release a bit too edgy for me.
ATM I'm pretty burned out by this mess and, because I don't have a script to reproduce consistently this issue, or the others which disappeared dropping wheels, I'm not sure if using 1.1 and de-deprecating the wheels is a good idea. So while all you wish are beautiful things to wish, I don't feel advocating for wheels anymore. I will still maintain them, as separate package, but I won't pour further research time in it.
You are welcome to contribute a solution of course, but it must be your time, not mine.
@dvarrazzo ...If I'm reading that commit you linked to correctly, it is explicitly disabling thread-safety in the psycopg2 wheels. So that would certainly explain the problems you're having.
It claims that it's enabling the libcrypto thread safety callbacks. The mechanism it uses to do this is to import the stdlib ssl module, and indeed, when ssl is imported then it will go and register the thread-safety callbacks. BUT... it does that in its copy of libcrypto. If libpq and ssl are both using the same copy of libcrypto, then this is fine. But in the psycopg2 wheels, they aren't. So after importing ssl, one copy of libcrypto has become thread-safe... but it's not the copy that libpq is using. And then that patch tells libpq, hey don't worry about thread-safety, we already took care of it.
End result: your openssl, in the wheel builds only, has no thread safety enabled. I suspect this explains at least a few of the weird issues people have been reporting; maybe all of them.
I totally get being burned out on this. I don't have the time to dig further either. But the good news it that if anyone reading this does want to try to revive the wheels, it looks like it might be much more doable than was previously believed to be the case :-). And BTW, the cryptography devs are shipping manylinux wheels with openssl 1.1 – if anyone wants to look into this further, they might be able to help you figure out how to build it on the manylinux container. (I suspect this is easier than trying to get the thread safety callbacks correct by hand.)
I'm happy to help fix the wheels, as above - where should I do a PR?
@matthew-brett That would be awesome!
cc @dvarrazzo
@matthew-brett I guess the https://github.com/psycopg/psycopg2-wheels package is the right place.
@rubendv - and others on this thread - we may well have a fix for the threading problems, in psycopg/psycopg2-wheels#8, but getting default wheels back, depends on having a test for this issue - see #837. We need a test case that we can run, that crashes using current psycopg2-binary. I'd really like a test database that I can put in the wheel repository for testing, or generate on the fly, but if you can't provide that, then a database that I can keep on my encrypted desktop drive, would be a lot better than nothing.