Factory generates same values for different test through one pytest run (Factory works only once).

[Kostiantyn-Salnykov]

Hi, I've created a Factory for the User (id => pk (unique), email => unique constraint), and provided __async_persistence__ to it.
But It works only once (first time, then all other time Faker generates the same values for id, and email, and maybe for other fields). Tried to fix it with Use declaration, but it helps only with id.

class BaseRawFactory(ModelFactory):
    @classmethod
    def get_mock_value(cls, field_type: Any) -> Any:
        type_name = str(field_type.__name__)
        if type_name == "Email":
            return cls.get_faker().email()

        return super().get_mock_value(field_type)


class BaseFactory(BaseRawFactory):
    created_at: datetime.datetime = PostGenerated(fn=generate_dt)
    updated_at: datetime.datetime = PostGenerated(fn=generate_dt)


class UserFactory(BaseFactory):
    """UserFactory based on Faker and Pydantic."""

    id = Use(fn=uuid.uuid4)  # MY HOTFIX FOR ID
    password_hash: str = PostGenerated(fn=make_password, password=DEFAULT_PASSWORD)

    __model__ = UserCreateToDBSchema
    __allow_none_optionals__ = False  # Factory will generate all fields (even for Optional fields)
    __async_persistence__ = AsyncPersistenceHandler(model=User)

I also used a faker_seed fixture in my conftest.py and faker fixture works as expected inside tests all the time.

@pytest.fixture(scope="function", autouse=True)
def faker_seed() -> int:
    """Generate random seed for Faker instance.

    Returns:
        Random generated integer from 0 up to 100000.
    """
    return random.randint(0, 100000)

It works as expected from IPython console:

await UserFactory.create_async()

It didn't work inside pytest tests ⚠️

(looks like inner __faker__ seed resets to same value on every test function run).
And actually, factory can be used only one time. The first test succeed, and the others - failed. With pytest-randomly this success & failed tests rotating randomly 🤷‍♂️.

My environment:
Python: 3.10
Test dependencies:

[tool.poetry.group.test.dependencies]
pytest = "^7.1.3"
pytest-asyncio = "^0.19.0"
pytest-mock = "^3.8.2"
pytest-sugar = "^0.9.5"
pytest-cov = "^3.0.0"
pytest-randomly = "^3.12.0"
pytest-clarity = "^1.0.1"
Faker = "^15.0.0"
pydantic-factories = "^1.7.0"
pytest-alembic = "^0.8.4"

Possible solutions (thought about it):

• Clear db after every test - this will slow down all tests;
• Rollback db session that used inside __async_persistence__ - this is hard to implement without the possibility to connect with pytest fixtures inside it.

Please provide any ideas on how to handle this behavior.

[jtraub]

@Kostiantyn-Salnykov could you please post a complete minimal example of model+factory+test so I can run it?

[Kostiantyn-Salnykov]

@jtraub it will be hard to extract all necessary source code, but sure that all related staff are located in tests/apps/users/test_models.py
tests/apps/users/factories.py
apps/users/models.py
apps/users/schemas.py
In the case of naming:
models - SQLAlchemy's models
schemas - Pydantic's models

Repo: https://github.com/Kostiantyn-Salnykov/fastapi_quickstart
This is poetry based repo (+ possible to use docker-compose) and exists command make test | make test_cov to run tests.

[jtraub]

@Kostiantyn-Salnykov thanks a lot for the code.

I will try to fix the problem in the next few days.

[jtraub]

@Kostiantyn-Salnykov the reason of your problem is pytest-randomly resetting random.seed for each test.

So you need to pass --randomly-dont-reset-seed flag if you want to get rid of this behaviour or get rid of pytest-randomly itself.

This command works fine for me poetry run pytest --test-alembic --randomly-dont-reset-seed

P.S. I would also recommend you to provide .env.example file in your repo so user can copy it to .env file and launch your project right away.

[jtraub]

As README.md in https://github.com/pytest-dev/pytest-randomly states

...
If faker is installed, its random state is reset at the start of every test. This is also for repeatable fuzzy data in tests - factory boy uses faker for lots of data. This is also done if you're using the faker pytest fixture, by defining the faker_seed fixture (docs).
...

it is the desired (default) behaviour.

So I would suggest to use --randomly-dont-reset-seed flag (so you get random test order from pytest-randomly).

If you need to get rid of randomness everywhere except Faker - that is a different story though.