testcontainers/testcontainers-python

TypeError: object.__new__() takes exactly one argument (the type to instantiate)

covatic-john opened this issue · 21 comments

covatic-john commented

Describe the bug

running testcontainers in a pytest using a with statement I get the following error

@wait_container_is_ready()
def _connect(self) -> MongoClient:

  return MongoClient(self.get_connection_url())

E TypeError: object.new() takes exactly one argument (the type to instantiate)

Just wondering if anyone else has encountered this

alexanderankin commented

can you post a little bit more details on how you got this output? some of your surrounding code maybe?

covatic-john commented

I'll update with test and example once I finish the urgent work for work.
the fix was simple just needed the self.get_connection_url() added to the init rather than the constructor

@wait_container_is_ready()
def _connect(self) -> MongoClient:
mc = MongoClient()
mc.init(self.get_connection_url())
return mc`

covatic-john commented

This error goes away with 4.0.0rc2

alexanderankin commented

Awesome, thanks for testing it! @covatic-john

covatic-john commented

Sorry @alexanderankin I hadn't cleared cache and this is still happening. I'll write test case for this and the localstack bug.

alexanderankin commented

if you have a stack trace for this that would also be helpful, I remember looking at this one and the mongodb client does not throw this error in the constructor, so its not obvious to me where it comes from.

if the bug is in the testcontainers library, presumably it is because we are not waiting for the container correctly or something like that. to be clear your code example is actually this, right:

@wait_container_is_ready()
def _connect(self) -> MongoClient:
    mc = MongoClient()
    mc.__init__(self.get_connection_url())
    return mc`

i made it syntax highlight by using this in my markdown:

```python
# code goes here
```
covatic-john commented
covatic-john commented

OK with 4.0.0rc2 I still get

details 
Container started: 731bdd5d5ae4
Waiting for container <Container: 731bdd5d5ae4> with image localstack/localstack:2.0.1 to be ready ...

test setup failed
@pytest.fixture(scope='session', autouse=True)
    def localstack():
        with LocalStackContainer(
                image="localstack/localstack:2.0.1"
        ).with_bind_ports(
            4566, 4566) as localstack:
            resp = urllib.request.urlopen(f"{localstack.get_url()}/health")
            services = json.loads(resp.read().decode())["services"]
            # Check that all services are running
            # assert all(value == "available" for value in services.values())
            # Check that some of the services keys
            assert all(test_service in services for test_service in ["dynamodb", "sns", "sqs"])
>           s3 = localstack.get_client("s3")
E           AttributeError: 'LocalStackContainer' object has no attribute 'get_client'

integration/test_pipelines.py:229: AttributeError

there is a fix for this already submitted. #372

running with 4.0.0.rc2 with patched localstack I get the following error with mongo

-------------------------------- live log setup --------------------------------
2024-02-14 14:25:59 [    INFO] Pulling image localstack/localstack:2.0.1 (container.py:60)
2024-02-14 14:26:00 [    INFO] Container started: ac4f215217f6 (container.py:66)
2024-02-14 14:26:01 [    INFO] Waiting for container <Container: ac4f215217f6> with image localstack/localstack:2.0.1 to be ready ... (waiting_utils.py:52)
2024-02-14 14:26:01 [    INFO] Waiting for container <Container: ac4f215217f6> with image localstack/localstack:2.0.1 to be ready ... (waiting_utils.py:52)
2024-02-14 14:26:02 [    INFO] Pulling image mongo:latest (container.py:60)
ERROR                                                                    [ 92%]Pulling image localstack/localstack:2.0.1
Container started: ac4f215217f6
Waiting for container <Container: ac4f215217f6> with image localstack/localstack:2.0.1 to be ready ...
Waiting for container <Container: ac4f215217f6> with image localstack/localstack:2.0.1 to be ready ...
Pulling image mongo:latest

test setup failed
@pytest.fixture(scope='session', autouse=True)
    def mongo_client():
>       with MongoDbContainer("mongo:latest", username='admin', password='admin').with_bind_ports(
                27017, 27020) as mongo_client1:

integration/test_pipelines.py:212: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/lib/python3.10/site-packages/testcontainers/core/container.py:73: in __enter__
    return self.start()
../venv/lib/python3.10/site-packages/testcontainers/core/generic.py:57: in start
    super().start()
../venv/lib/python3.10/site-packages/testcontainers/core/container.py:62: in start
    self._container = docker_client.run(
../venv/lib/python3.10/site-packages/testcontainers/core/docker_client.py:50: in run
    container = self.client.containers.run(
../venv/lib/python3.10/site-packages/docker/models/containers.py:873: in run
    container = self.create(image=image, command=command,
../venv/lib/python3.10/site-packages/docker/models/containers.py:931: in create
    create_kwargs = _create_container_args(kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

kwargs = {'password': 'admin', 'username': 'admin'}

    def _create_container_args(kwargs):
        """
        Convert arguments to create() to arguments to create_container().
        """
        # Copy over kwargs which can be copied directly
        create_kwargs = {}
        for key in copy.copy(kwargs):
            if key in RUN_CREATE_KWARGS:
                create_kwargs[key] = kwargs.pop(key)
        host_config_kwargs = {}
        for key in copy.copy(kwargs):
            if key in RUN_HOST_CONFIG_KWARGS:
                host_config_kwargs[key] = kwargs.pop(key)
    
        # Process kwargs which are split over both create and host_config
        ports = kwargs.pop('ports', {})
        if ports:
            host_config_kwargs['port_bindings'] = ports
    
        volumes = kwargs.pop('volumes', {})
        if volumes:
            host_config_kwargs['binds'] = volumes
    
        network = kwargs.pop('network', None)
        networking_config = kwargs.pop('networking_config', None)
        if network:
            if networking_config:
                # Sanity check: check if the network is defined in the
                # networking config dict, otherwise switch to None
                if network not in networking_config:
                    networking_config = None
    
            create_kwargs['networking_config'] = NetworkingConfig(
                networking_config
            ) if networking_config else {network: None}
            host_config_kwargs['network_mode'] = network
    
        # All kwargs should have been consumed by this point, so raise
        # error if any are left
        if kwargs:
>           raise create_unexpected_kwargs_error('run', kwargs)
E           TypeError: run() got unexpected keyword arguments 'password', 'username'

../venv/lib/python3.10/site-packages/docker/models/containers.py:1159: TypeError

If I remove the username and password arguments I am then left with the original error

-------------------------------- live log setup --------------------------------
2024-02-14 14:29:04 [    INFO] Pulling image localstack/localstack:2.0.1 (container.py:60)
2024-02-14 14:29:04 [    INFO] Container started: 298bdc4bd535 (container.py:66)
2024-02-14 14:29:05 [    INFO] Waiting for container <Container: 298bdc4bd535> with image localstack/localstack:2.0.1 to be ready ... (waiting_utils.py:52)
2024-02-14 14:29:06 [    INFO] Waiting for container <Container: 298bdc4bd535> with image localstack/localstack:2.0.1 to be ready ... (waiting_utils.py:52)
2024-02-14 14:29:06 [    INFO] Pulling image mongo:latest (container.py:60)
2024-02-14 14:29:06 [    INFO] Container started: b176a12044af (container.py:66)
2024-02-14 14:29:06 [    INFO] Waiting for container <Container: b176a12044af> with image mongo:latest to be ready ... (waiting_utils.py:52)
2024-02-14 14:29:06 [    INFO] Waiting for container <Container: b176a12044af> with image mongo:latest to be ready ... (waiting_utils.py:52)
ERROR                                                                    [ 92%]Pulling image localstack/localstack:2.0.1
Container started: 298bdc4bd535
Waiting for container <Container: 298bdc4bd535> with image localstack/localstack:2.0.1 to be ready ...
Waiting for container <Container: 298bdc4bd535> with image localstack/localstack:2.0.1 to be ready ...
Pulling image mongo:latest
Container started: b176a12044af
Waiting for container <Container: b176a12044af> with image mongo:latest to be ready ...
Waiting for container <Container: b176a12044af> with image mongo:latest to be ready ...

test setup failed
@pytest.fixture(scope='session', autouse=True)
    def mongo_client1):
>       with MongoDbContainer("mongo:latest").with_bind_ports(
                27017, 27020) as mongo_client1:

integration/test_pipelines.py:212: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/lib/python3.10/site-packages/testcontainers/core/container.py:73: in __enter__
    return self.start()
../venv/lib/python3.10/site-packages/testcontainers/core/generic.py:58: in start
    self._connect()
../venv/lib/python3.10/site-packages/testcontainers/core/waiting_utils.py:60: in wrapper
    return wrapped(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testcontainers.mongodb.MongoDbContainer object at 0x29406b1c0>

    @wait_container_is_ready()
    def _connect(self):
        from pymongo import MongoClient
>       return MongoClient(self.get_connection_url())
E       TypeError: object.__new__() takes exactly one argument (the type to instantiate)

../venv/lib/python3.10/site-packages/testcontainers/mongodb/__init__.py:78: TypeError

This will fail running as pytest. if you patch localstack with get_client() method then you will get a username and password error and then if you remove the username and password you will get the original error above

import json
import urllib

import pytest
from testcontainers.localstack import LocalStackContainer
from testcontainers.mongodb import MongoDbContainer

MONGO_LATEST = "mongo:latest"

S3_FOLDER = "test-bucket"


def test_client_and_brand(mongo_client, mongo_london, mongo_client2, mongo_client1, localstack):
    assert localstack.get_url() is not None
    assert mongo_client.get_connection_url() is not None
    assert mongo_london.get_connection_url() is not None
    assert mongo_client2.get_connection_url() is not None
    assert mongo_client1.get_connection_url() is not None


@pytest.fixture(scope='session', autouse=True)
def mongo_client():
    with MongoDbContainer(MONGO_LATEST, username='admin', password='admin').with_bind_ports(
            27017, 27017) as mongo_client:
        yield mongo_client


@pytest.fixture(scope='session', autouse=True)
def mongo_london():
    with MongoDbContainer(MONGO_LATEST, username='admin', password='admin').with_bind_ports(
            27017, 27018) as mongo_london:
        yield mongo_london


@pytest.fixture(scope='session', autouse=True)
def mongo_client2():
    with MongoDbContainer(MONGO_LATEST, username='admin', password='admin').with_bind_ports(
            27017, 27019) as mongo_client2:
        yield mongo_client2


@pytest.fixture(scope='session', autouse=True)
def mongo_client1():
    with MongoDbContainer("mongo:latest").with_bind_ports(
            27017, 27020) as mongo_client1:
        yield mongo_client1


@pytest.fixture(scope='session', autouse=True)
def localstack():
    with LocalStackContainer(
            image="localstack/localstack:2.0.1"
    ).with_bind_ports(
        4566, 4566) as localstack:
        resp = urllib.request.urlopen(f"{localstack.get_url()}/health")
        services = json.loads(resp.read().decode())["services"]
        # Check that all services are running
        # assert all(value == "available" for value in services.values())
        # Check that some of the services keys
        assert all(test_service in services for test_service in ["dynamodb", "sns", "sqs"])
        s3 = localstack.get_client("s3")
        s3.create_bucket(
            Bucket=S3_FOLDER,
            CreateBucketConfiguration={"LocationConstraint": "us-west-1"},
        )
        yield localstack

The fix for mongodb is

mongodb/testcontainers/mongodb/init.py

    @wait_container_is_ready()
    def _connect(self) -> MongoClient:
        mc = MongoClient()
        mc.__init__(self.get_connection_url())
        return mc
covatic-john commented

@alexanderankin hope that covers it :). Now going to see if the docker issue has been fixed in RC2 as I have patched core as well

covatic-john commented

fix for testcontainers-localstack is in main branch but there has not been a release for the feature

totallyzen commented

@covatic-john we've released 4.0 🎉 can we then close this issue? Has everything been solved for you?

covatic-john commented
totallyzen commented

Does 4.0 include updates on the features, localstack[mongodb} or Localstack? I’ll test this morning

Thanks for the help! Not enough time to fix/test everything personally. 💚

covatic-john commented

4.0.0 tested Still not fixed needs a release of testcontainers-mongodb
pypi still has 0.0.1rc1. will try installing from main branch

alexanderankin commented
covatic-john commented

@alexanderankin sorry was busy last week with a P1 and migrations. merging and release would be fantastic 👍

Localstack giving me no issue now with new release :)

alexanderankin commented
alexanderankin commented

4.0.1 is out (maybe at the expense of our release-please pipeline)

can install testcontainers[mongodb] and see if anything is fixed if not then #448 is back on the table

covatic-john commented

nope still same error

integration/test_pipelines.py:53: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
integration/helpers.py:137: in setup_mongo_clients_aws_s3
    london_db = mongo_london.get_connection_client().serendipity_dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testcontainers.mongodb.MongoDbContainer object at 0x28c9cd0c0>

    def get_connection_client(self) -> MongoClient:
>       return MongoClient(self.get_connection_url())
E       TypeError: object.__new__() takes exactly one argument (the type to instantiate)

../venv/lib/python3.10/site-packages/testcontainers/mongodb/__init__.py:88: TypeError
covatic-john commented

Tested this with the latest release still not working, this fixes the issue it's to do with how you are initialising the mongoclient

I know not pythonistic but it works :)

    def get_connection_client(self) -> MongoClient:
        mc = MongoClient()
        mc.__init__(self.get_connection_url())
        return mc
santi commented

Merged another fix here #448, please check it out when it is released in the next package version.

When that happens, please provide:

  • OS and version / build number
  • Testcontainers version. Do not blindly trust the version you specify in your setup. Check it programmatically with a short snippet of
from importlib.metadata import version
print(version("testcontainers"))