Setting up responses for classes with unittest yeilds two different registeries

[Seluj78]

Describe the bug

I am using unittest along with responses to test my Flask API. I use responses to mock the requests made to other servers, for example to send an email. I am also using moto for other mocking reasons.

My tests are configured with a _TestCase which, stripped down to just what matters to responses, looks like this:

class _TestCase(unittest.TestCase):
    def setUp(self):
        self.r_mock = responses.RequestsMock(assert_all_requests_are_fired=True)
        self.r_mock.start()

    def tearDown(self):
        self.r_mock.stop()
        self.r_mock.reset()

When running tests classes (located in different files, that inherit from _TestCase), I get a weird behavior where the responses are registered in the self.r_mock registery but ignored. My API only uses the ones I register with responses.add and not responses.add (which also bypasses the assert_all_requests_are_fired)

I'm at a loss here 🙏

Additional context

An example test class would look something like

@mock_dynamodb
@mock_ec2
class TestDeploymentNewUser(_TestCase):
    def setUp(self):
        super().setUp()
        self.r_mock.reset()
		# Other code used to create stuff in dynamodb etc

When running in debug mode, here are the registered() of self.r_mock and responses at different point in my test

At the start of the test method, before anything is executed in the test

>>> self.r_mock.registered()
[]
>>> responses.registered()
[]

After registering some self.r_mock.add calls

>>> self.r_mock.registered()
[<Response(url='https://xxx.email/user/kickoff' status=200 content_type='text/plain' headers='null')>, <Response(url='https://xxx.email/user/credentials' status=200 content_type='text/plain' headers='null')>, <Response(url='re.compile('http://.*/create/\\d+')' status=200 content_type='text/plain' headers='null')>, <Response(url='re.compile('http://localhost:8080/api/xxx/\\d+')' status=500 content_type='text/plain' headers='null')>]
>>> responses.registered()
[]

Moving through the test until before I call the test client to make a request to test my code, it's exactly the same thing.
Now, With my debugger, I enter the code being executed when the API is called by my test and I reach the first call that I want mocked. self.r_mock doesn't exist in this context because we're in a completely different code, but I can import responses and I get this (cut for clarity):

>>> responses.registered()
[<moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5650>, <moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5810>, <moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5990>, '...']

Using the PyCharm debugger, I can move back through the thread back to my unit test where my test code is awaiting a response and I get this:

>>> self.r_mock.registered()
[<Response(url='https://xxx.email/user/kickoff' status=200 content_type='text/plain' headers='null')>, <Response(url='https://xxx.email/user/credentials' status=200 content_type='text/plain' headers='null')>, <Response(url='re.compile('http://.*/create/\\d+')' status=200 content_type='text/plain' headers='null')>, <Response(url='re.compile('http://localhost:8080/api/xxx/\\d+')' status=500 content_type='text/plain' headers='null')>]
>>> responses.registered()
[<moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5650>, <moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5810>, <moto.core.custom_responses_mock.CallbackResponse object at 0x120ce5990>, '...']

And, if I step through, the requests.post are executed and not mocked because they aren't in responses.registered but in self.r_mock.registered().

Version of responses

0.23.3

Steps to Reproduce

See above :)

Expected Result

>>> self.r_mock.registered()
[<Response(url='re.compile('http://localhost:8080/api/xxx/\\d+')' status=500 content_type='text/plain' headers='null')>, 'A lot of moto Callbacks']

>>> responses.registered()
[]

Actual Result

>>> self.r_mock.registered()
[<Response(url='re.compile('http://localhost:8080/api/xxx/\\d+')' status=500 content_type='text/plain' headers='null')>]

>>> responses.registered()
['A lot of moto callbacks']

[bblommers]

@Seluj78 If I understand it correctly, the root cause is that Moto has no way to retrieve the existing responses, so it will always override them with it's own.

This has been raised in Moto as well, with a possible workaround: getmoto/moto#6417

[Seluj78]

Hmm, I didn't think moto would be the core of the problem, but I had it included since it was showing up a lot in the registered responses.

My main problem here is the fact that my self.r_mock responses are ignored and only the ones in responses are accepted. I will try responses._real_send = rsps.unbound_on_send() in my codebase and see if it changes anything.

[Seluj78]

Although in the example, it uses rsps which is from the with statement, which I do not have in my code. I will try with self.r_mock but if it doesn't work, I suppose I need to use the rsps from moto ? it seems like it worked

[Seluj78]

Ok, it seemed to have worked immediately. Further investigation needed on more tests 👀

[Seluj78]

Yeah it's half working. With a passthrough, I'm getting a recursion error.

Inside the setUp

        self.r_mock = responses.RequestsMock(assert_all_requests_are_fired=True)
        self.r_mock.start()
        self.r_mock.add_passthru(re.compile(rf"{MEILISEARCH_URL}.*"))
        responses._real_send = self.r_mock.unbound_on_send()

exception log:

hub_backend/routes/api/admin/deployment.py:292: in deploy_user
    index_cohort(cohort)
hub_backend/utils/meilisearch.py:57: in index_cohort
    cohort_index.add_documents(
.venv/lib/python3.11/site-packages/meilisearch/index.py:381: in add_documents
    add_document_task = self.http.post(url, documents)
.venv/lib/python3.11/site-packages/meilisearch/_httprequests.py:73: in post
    return self.send_request(requests.post, path, body, content_type)
.venv/lib/python3.11/site-packages/meilisearch/_httprequests.py:51: in send_request
    request = http_method(
.venv/lib/python3.11/site-packages/requests/api.py:115: in post
    return request("post", url, data=data, json=json, **kwargs)
.venv/lib/python3.11/site-packages/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
.venv/lib/python3.11/site-packages/requests/sessions.py:587: in request
    resp = self.send(prep, **send_kwargs)
.venv/lib/python3.11/site-packages/requests/sessions.py:701: in send
    r = adapter.send(request, **kwargs)
.venv/lib/python3.11/site-packages/responses/__init__.py:1127: in send
    return self._on_request(adapter, request, **kwargs)
.venv/lib/python3.11/site-packages/responses/__init__.py:1033: in _on_request
    return _real_send(adapter, request, **kwargs)
.venv/lib/python3.11/site-packages/responses/__init__.py:1127: in send
    return self._on_request(adapter, request, **kwargs)
.venv/lib/python3.11/site-packages/responses/__init__.py:1033: in _on_request
    return _real_send(adapter, request, **kwargs)
.venv/lib/python3.11/site-packages/responses/__init__.py:1127: in send
    return self._on_request(adapter, request, **kwargs)
E   RecursionError: maximum recursion depth exceeded
!!! Recursion detected (same locals & position)

[Seluj78]

Every single test in my test suite that calls a route that uses requests fails with a RecursionError.

[beliaev-maksim]

@Seluj78 it looks like original issue is solved

But now you hit another one, would you mind open another bug and fill the details that will help us to debug

[Seluj78]

Makes sense. Should I close this one and open another one in getsentry/responses ? Or do you think this has more to do with moto still ?

[beliaev-maksim]

If you can provide a reproducible without moto, then we can have a look in responses

Then please close current issue

[Seluj78]

Closing this, new issue is #670