“AttributeError: ‘_thread._local’ object has no attribute ‘no_dereferencing_class’ in Multithreaded Environment”

[AdithyanJothir]

Hi,

Details :

In my project, I’m using ASGI’s sync_to_async utility. I believe this utility creates multiple threads using a thread pool executor, and each thread has its own thread-local storage. The attribute no_dereferencing_class, set by MongoEngine in context_managers.py, is causing an error: ‘_thread._local’ object has no attribute ‘no_dereferencing_class’. This issue is observed in the context_managers.py file.

thread_locals = threading.local()
thread_locals.no_dereferencing_class = {}


def no_dereferencing_active_for_class(cls):
    return cls in thread_locals.no_dereferencing_class

Tested on versions:

• Python 3.12.1
• MongoEngine 0.28.0
• PyMongo 4.6.2

[bagerard]

Argh this change was brought in the last release.
Could you provide a reproducible snippet containing minimal amount of code?

[kelvin-j-li]

I have the same error with eventlet-0.35.2 when running mongoengine-0.28.0 and flask_track_usage-2.0.0.

Fallback to mongoengine-0.27.0 help remove the no_dereferencing_class error:

  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/__init__.py", line 53, in _caller
    method(**kwargs)
  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/mongoenginestorage.py", line 139, in sumUrl
    increment(sumUrlClasses, src, "url", ["url"])
  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/mongoenginestorage.py", line 38, in increment
    doc = class_dict[period].objects(date=times[period], **db_args).first()
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 301, in first
    result = queryset[0]
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 206, in __getitem__
    _auto_dereference=self._auto_dereference,
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 1768, in _auto_dereference
    should_deref = not no_dereferencing_active_for_class(self._document)
  File "/home/pyuser/app/dist-packages/mongoengine/context_managers.py", line 28, in no_dereferencing_active_for_class
    return cls in thread_locals.no_dereferencing_class
  File "/home/pyuser/app/dist-packages/eventlet/corolocal.py", line 45, in __getattribute__
    return object.__getattribute__(self, attr)
AttributeError: 'local' object has no attribute 'no_dereferencing_class'

Thanks!

[kelvin-j-li]

# activate eventlet
import os
import eventlet
eventlet.monkey_patch()

from flask_track_usage.storage.mongo import MongoEngineStorage
from mongoengine import connect

[saipraveenkondapalli]

I am facing the same issue after updating to the latest version 0.28.0. The issue occurs only in the latest version, downgrading to version 0.27.0 seems to solve the issue.

sample code from my project

@auth_bp.route('/login', methods=['POST'])
def login():
    data = request.get_json()
    email = data.get('email')
    password = data.get('password')
    user = User.objects(email=email).first()
    if user and check_password_hash(user.password, password):
        return make_response('Logged in', 200)
    return make_response('Invalid email or password', 401)

The issue occurs where ever there is a query in the code.

[AdithyanJothir]

Here is a simple code snippet to reproduce the said issue using asgiref sync to async with ThreadPoolExecuter

import asyncio
from concurrent.futures import ThreadPoolExecutor
from functools import partial
import mongoengine
from asgiref.sync import sync_to_async
from mongoengine import Document, StringField

mongoengine.connect(
    host="mongodb://localhost:27017/",
    db="test_db",
)


class UserModel(Document):
    meta = {
        "collection": "users",
    }
    name = StringField()


user_a = UserModel(name="JothirAdithyan")
UserModel.objects.insert(user_a)


async def get_docs():
    function = partial(UserModel.objects.first)
    user1 = await sync_to_async(func=function, thread_sensitive=False, executor=ThreadPoolExecutor())()
    print(user1.to_json())

if __name__ == "__main__":
    asyncio.run(get_docs())

[c4talyst]

Here is an example using just Flask (no async).

tl;dr: Downgrade to mongoengine 0.27.0 and it works

requirements.txt

Flask==3.0.2
mongoengine==0.28.0

app.py

app = Flask(__name__)

from flask import Flask
from mongoengine import connect
from mongoengine import Document, StringField

app = Flask(__name__)
app.config['MONGODB_SETTINGS'] = {
    'db': 'my_database',
    'host': 'mongodb://localhost:20000/my_database'
}
connect(db='my_database', host='mongodb://localhost:20000/my_database')

class User(Document):
    user_id = StringField(required=True)
    name = StringField()
    email = StringField(required=True)

@app.route('/')
def index():
    # Create a new user
    user = User(user_id='john_doe', email='john@example.com')
    user.save()
    return 'User created successfully!'

@app.route('/users')
def get_users():
    # Retrieve all users
    users = User.objects.all()
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
    return f'Users: {user_list}'


if __name__ == "__main__":
    app.run(debug=True, port=6001)

Exception

Traceback (most recent call last):
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1488, in __call__
    return self.wsgi_app(environ, start_response)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1466, in wsgi_app
    response = self.handle_exception(e)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/tmp/temp-flask-mongo/app.py", line 32, in get_users
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
  File "/tmp/temp-flask-mongo/app.py", line 32, in <listcomp>
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/queryset.py", line 109, in _iter_results
    self._populate_cache()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/queryset.py", line 128, in _populate_cache
    self._result_cache.append(next(self))
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/base.py", line 1636, in __next__
    _auto_dereference=self._auto_dereference,
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/base.py", line 1768, in _auto_dereference
    should_deref = not no_dereferencing_active_for_class(self._document)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/context_managers.py", line 28, in no_dereferencing_active_for_class
    return cls in thread_locals.no_dereferencing_class
AttributeError: '_thread._local' object has no attribute 'no_dereferencing_class'

Fix

Downgrade mongoengine to 0.27.0

[HK-Mattew]

Recently I'm getting the same error

[bagerard]

Thanks for the reproducible snippet, working on a fix right now

[jan-stillfried-toteat]

We're facing the same issue. Downgrading fixed it for now.

[voglster]

Just hit this myself. Looks like the patch is on the way

[bagerard]

merged and 0.28.1 will be published in a few minutes