Error during stress testing

[carlosribas]

I submitted 30 jobs almost at the same time (using a pause of 0.1 sec) and got this error:

WARNING:aiopg:Invalid transaction status on released connection: 1
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<check_chunks_and_consumers() done, defined at /srv/sequence_search/producer/main.py:54> exception=DatabaseConnectionError('Connection closed',)>
Traceback (most recent call last):
File "/srv/sequence_search/db/consumers.py", line 49, in find_available_consumers
async for row in connection.execute(query, status=CONSUMER_STATUS_CHOICES.available):
File "/usr/local/lib/python3.6/site-packages/aiopg/utils.py", line 94, in anext
self._obj = yield from self._coro
File "/usr/local/lib/python3.6/site-packages/aiopg/sa/connection.py", line 110, in _execute
yield from cursor.execute(str(compiled), post_processed_params[0])
File "/usr/local/lib/python3.6/site-packages/aiopg/cursor.py", line 118, in execute
yield from self._conn._poll(waiter, timeout)
File "/usr/local/lib/python3.6/site-packages/aiopg/connection.py", line 244, in _poll
yield from asyncio.wait_for(self._waiter, timeout, loop=self._loop)
File "/usr/lib64/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
psycopg2.OperationalError: Connection closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/srv/sequence_search/producer/main.py", line 57, in check_chunks_and_consumers
available_consumers = await find_available_consumers(app['engine'])
File "/srv/sequence_search/db/consumers.py", line 55, in find_available_consumers
raise DatabaseConnectionError(str(e)) from e
sequence_search.db.DatabaseConnectionError: Connection closed
ERROR:aiohttp.server:Error handling request
Traceback (most recent call last):
File "/srv/sequence_search/db/job_chunks.py", line 70, in save_job_chunk
status=JOB_CHUNK_STATUS_CHOICES.pending
File "/usr/local/lib/python3.6/site-packages/aiopg/sa/connection.py", line 121, in scalar
res = yield from self.execute(query, *multiparams, **params)
File "/usr/local/lib/python3.6/site-packages/aiopg/utils.py", line 66, in iter
resp = yield from self._coro
File "/usr/local/lib/python3.6/site-packages/aiopg/sa/connection.py", line 110, in _execute
yield from cursor.execute(str(compiled), post_processed_params[0])
File "/usr/local/lib/python3.6/site-packages/aiopg/cursor.py", line 108, in execute
waiter = self._conn._create_waiter('cursor.execute')
File "/usr/local/lib/python3.6/site-packages/aiopg/connection.py", line 218, in _create_waiter
'data' % func_name)
RuntimeError: cursor.execute() called while another coroutine is already waiting for incoming data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/aiohttp/web_protocol.py", line 418, in start
resp = await task
File "/usr/local/lib64/python3.6/site-packages/aiohttp/web_app.py", line 458, in _handle
resp = await handler(request)
File "/usr/local/lib64/python3.6/site-packages/aiohttp/web_middlewares.py", line 119, in impl
return await handler(request)
File "/usr/local/lib64/python3.6/site-packages/aiohttp/web_middlewares.py", line 109, in impl
return await handler(request)
File "/usr/local/lib/python3.6/site-packages/aiojobs/aiohttp.py", line 39, in wrapper
return await job.wait()
File "/usr/local/lib/python3.6/site-packages/aiojobs/_job.py", line 64, in wait
await self._close(scheduler.close_timeout)
File "/usr/local/lib/python3.6/site-packages/aiojobs/_job.py", line 89, in _close
await self._task
File "/usr/local/lib/python3.6/site-packages/aiojobs/_job.py", line 59, in wait
loop=self._loop)
File "/usr/local/lib/python3.6/site-packages/aiojobs/_job.py", line 50, in _do_wait
return await self._task
File "/srv/sequence_search/producer/views/submit_job.py", line 112, in submit_job
job_chunk_id = await save_job_chunk(request.app['engine'], job_id, database)
File "/srv/sequence_search/db/job_chunks.py", line 76, in save_job_chunk
"job_id = %s, database = %s" % (job_id, database)) from e
sequence_search.db.SQLError: Failed to save job_chunk for job_id = f9b6d65d-2d06-4f15-8a8a-8ab866a0acaf, database = whitelist-rrna.part_007.fasta

I need to investigate the problem and check the way that I'm testing the server. This error does not happen often.