Dashboard: pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.

[tfriedel]

I'm getting this error when I try to browse the "statuses" page in the dashboard and I have a large (~ 200.000) number of jobs.

172.17.0.1 - - [17/Feb/2021 17:13:11] "GET /api/datatables/status?sEcho=1&iColumns=4&sColumns=&iDisplayStart=0&iDisplayLength=20&mDataProp_0=function&mDataProp_1=function&mDataProp_2=function&mDataProp_3=function&sSearch=&bRegex=false&sSearch_0=&bRegex_0=false&bSearchable_0=true&sSearch_1=&bRegex_1=false&bSearchable_1=true&sSearch_2=&bRegex_2=false&bSearchable_2=true&sSearch_3=&bRegex_3=false&bSearchable_3=true&iSortCol_0=0&sSortDir_0=asc&iSortingCols=1&bSortable_0=true&bSortable_1=true&bSortable_2=true&bSortable_3=true HTTP/1.1" 500 -
Error on request:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 304, in run_wsgi
    execute(self.server.app)
  File "/usr/local/lib/python3.6/dist-packages/werkzeug/serving.py", line 292, in execute
    application_iter = app(environ, start_response)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2463, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2449, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1866, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2446, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1951, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1820, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.6/dist-packages/mrq/dashboard/app.py", line 82, in api_jobstatuses
    {"$group": {"_id": "$status", "jobs": {"$sum": 1}}}
  File "/usr/local/lib/python3.6/dist-packages/mrq/monkey.py", line 99, in mrq_monkey_patched
    ret = base_method(self, *args, **kwargs)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/collection.py", line 2458, in aggregate
    **kwargs)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/collection.py", line 2377, in _aggregate
    retryable=not cmd._performs_write)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1471, in _retryable_read
    return func(session, server, sock_info, slave_ok)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/aggregation.py", line 148, in get_cursor
    user_fields=self._user_fields)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/pool.py", line 694, in command
    exhaust_allowed=exhaust_allowed)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/network.py", line 161, in command
    parse_write_concern_error=parse_write_concern_error)
  File "/home/thomas/.local/lib/python3.6/site-packages/pymongo/helpers.py", line 160, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting., full error: {'ok': 0.0, 'errmsg': 'Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting.', 'code': 16819, 'codeName': 'Location16819'}

[tfriedel]

I may have resolved this by running
mrq-run mrq.basetasks.indexes.EnsureIndexes
because I definitely didn't have any indexes in mongodb. I first had to also change MONGODB_LOGS = "1" to something else instead of 1 (actual URI), because otherwise I would get this error:

2021-02-17 18:45:52,957 [DEBUG] Starting mrq.basetasks.indexes.EnsureIndexes({})
2021-02-17 18:45:52,958 [ERROR] Job failed
2021-02-17 18:45:52,959 [ERROR] Traceback (most recent call last):
  File "/home/thomas/git/mrq_dashboard/venv/lib/python3.6/site-packages/mrq/worker.py", line 680, in perform_job
    job.perform()
  File "/home/thomas/git/mrq_dashboard/venv/lib/python3.6/site-packages/mrq/job.py", line 292, in perform
    result = self.task.run_wrapped(self.data["params"])
  File "/home/thomas/git/mrq_dashboard/venv/lib/python3.6/site-packages/mrq/task.py", line 20, in run_wrapped
    return self.run(params)
  File "/home/thomas/git/mrq_dashboard/venv/lib/python3.6/site-packages/mrq/basetasks/indexes.py", line 10, in run
    connections.mongodb_logs.mrq_logs.ensure_index(
AttributeError: 'int' object has no attribute 'mrq_logs'