End-to-end testing setup for Cap website
sabzo opened this issue · 3 comments
PR #2086
In order to better maintain CAP I've begun to setup End to End testing. I've been exploring multiple ways:
End to End Tests in same container
Goal is to add pytest-playwright into requirements.in
in order to auto-generated a requirements.txt which supports pytest-playwright. This requirements.txt is used in the Dockerfile that build the "web" image.
To generate a new requirements.txt one has to be in a python environment that has Fabric installed. Since I already have a local CAP installation, which has a python environment running, I decided to use the "web" container.
Entering into the "web" container running and the using fab compile doesn't work:
When running fab pip compile
or fab pip-compile
(according to the fabfile.py) inside the web container in order to generate new requirements.txt that will include playwright, the following error shows:
Calling pip-compile --generate-hashes --allow-unsafe
Could not find a version that matches text-unidecode==1.2,>=1.3 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))
Tried: 0.1, 1.0, 1.0, 1.1, 1.1, 1.2, 1.2, 1.3, 1.3
There are incompatible versions in the resolved dependencies:
text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.3.0->-r requirements.in (line 75))
text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/fabric/main.py", line 763, in main
*args, **kwargs
File "/usr/local/lib/python3.7/site-packages/fabric/tasks.py", line 427, in execute
results['<local-only>'] = task.run(*args, **new_kwargs)
File "/usr/local/lib/python3.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/app/fabfile.py", line 94, in pip_compile
subprocess.check_call(command, env=dict(os.environ, CUSTOM_COMPILE_COMMAND='fab pip-compile'))
File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip-compile', '--generate-hashes', '--allow-unsafe']' returned non-zero exit status 2.
Somehow adding playwright to requirements.in (in my case this was line 75) it seems to break the text-unidecode dependency of the factory-boy library:
text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.3.0->-r requirements.in (line 75))
text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 68))
It seems Pip is unable to have two versions of the same library running at the same time.
Can we manualy hack and try to simulate the Fab process?
Adding playwright run command to the Dockerfile for web doesn't work if the requirements.txt hasn't been generated anew:
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 3/13] RUN apt-get update && apt-get install -y redis-server pos 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 4/13] RUN mkdir /app 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 5/13] WORKDIR /app 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 6/13] COPY requirements.txt /app 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 7/13] RUN pip install pip==21.3.1 && pip install -r requirement 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 8/13] RUN echo "--modules-folder /node_modules" > /.yarnrc 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 9/13] COPY package.json /app 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 10/13] COPY yarn.lock /app 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 11/13] RUN curl -o nodejs.deb https://deb.nodesource.com/node_14.x/p 0.0s
=> CACHED [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 12/13] RUN apt install --no-install-recommends libdbus-glib-1-2 0.0s
=> ERROR [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 13/13] RUN playwright install chromium firefox 0.3s
------
> [registry.lil.tools/harvardlil/cap-web:192-d381c43315239f14e2a78be3b1d99433 13/13] RUN playwright install chromium firefox:
#0 0.251 /bin/sh: 1: playwright: not found
Realizing that removing pytest-playwright
or factory-boy
allows fab pip-compile to generate a requirements.txt, I devised an ugly hack that manually adds the generated portion of the requirements.txt for both factory-boy and pytest-playwright. Rebuilding the image docker-compose up --build -d
only produced this error:
#0 71.15 The conflict is caused by:
#0 71.15 The user requested text-unidecode==1.3
#0 71.15 faker 1.0.2 depends on text-unidecode==1.2
#0 71.15
#0 71.15 To fix this you could try to:
#0 71.15 1. loosen the range of package versions you've specified
#0 71.15 2. remove package versions to allow pip attempt to solve the dependency conflict
The auto generated requirements.txt has one place where it explicitly mentioned text-unidecode
-- what if this place is changed manually to version 1.2 as opposed to 1.3 to ensure the same text-unicode library is used? This also requires changing the expected SHA (which one gets by first failing by using an incorrect SHA). This solves pytest-playwright - faker issue, but introduces another issue because another library also depends on text-unicode==1.3..
#0 75.00 The conflict is caused by:
#0 75.00 The user requested text-unidecode==1.2
#0 75.00 faker 1.0.2 depends on text-unidecode==1.2
#0 75.00 python-slugify 6.1.2 depends on text-unidecode>=1.3
Manually hacking an auto-generated process may help debugging, but it doesn't solve the core issue!
Back to square 1!
The perma way?
Since perma works and already has playwright tests, can Perma's requirements.in be copied and used within CAP?
Copying over Perma's playwright/pytest-playwright dependencies returns following error:
Could not find a version that matches text-unidecode==1.2,>=1.3 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 70))
Tried: 0.1, 1.0, 1.0, 1.1, 1.1, 1.2, 1.2, 1.3, 1.3
There are incompatible versions in the resolved dependencies:
text-unidecode>=1.3 (from python-slugify==6.1.2->pytest-playwright==0.2.2->-r requirements.in (line 67))
text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0->-r requirements.in (line 70))
This still didn't work! This is because factory-boy install was old, outdated to version factory-boy==2.12.0 which is from the 2013 release! How did docker-compose up ...
result in installing a library outdated by more than 9 years?
For some reason, factory-boy is being hardcoded to 2.12.0. If no version is specified in requirements.in then why is it being hardcoded? fab pip-compile
doesn't generate an updated version of factoy-boy
locally or on the requirements files uploaded to Github. This is undefined behavior. At the same time the Factory library is outdated and
requirements.in intentionally uses the outdated Fabric library known as Fabric 3.
Fabric 3 is totally unsupported and leaves CAP vulnerable, not to mention it creates weird dependency issues when pytest-playwright is installed.
Perma uses the outdated Fabric3 but the dependency issues don't come up as Perma doesn't use the factory-boy library. The outdated libraries in Perma become a nuance when pytest-playwright and factory-boy is introduced for mocking as they require different versions of the same library.
Maybe the current web-image has some issues, what about using a brand new Python 3 install?
If fab pip-compile
within the CAP docker-compose environment, won't generate an updated requirements.txt where factory-boy is updated, can a clean new python3 installation work?
Another attempt was to run docker python install and create a virtual directory mapping: docker run -it -v $PWD:/app python:3.10.2 /bin/bash
so that fab pip compile
would generate a _requirements.txt file in the host but that didn't work either. Instead the error produced was:
File "/app/fabfile.py", line 15, in <module>
import django
ModuleNotFoundError: No module named 'django'
Since the new python container didn't have django pre-installed. Installing the missing libraries in the import statements in fabfile.py worked, however the fabfile.py has deprecated code that doesn't work in Pthon3. Although the Dockerfile for CAP uses Python3.7 it's unclear how fabric.api could've worked when it's unsupported
https://www.fabfile.org/upgrading.html#api-organization
Updating factory-boy itself may provide a path towards a solution solution...factory boy got updated but Faker remained at 1.0.2 (current version is 13.15.1). So pip uinstall faker
and then reinstalling factory-boy pip install factory-boy
updated the faker dependency to 13.15.1
This still leaves the problem of Fabric being outdated an having to be changed, in particular how we currently use local to look more like this.
TODO
- Update Fabric to the modern supported version
- Prevent
fab pip-compile
from hardcoding outdatedfactory-boy
version. - Change outdated fabric functions to new version
- Ensure factory-boy and dependencies are modern enough not to clash with
pytest-playwright
End to End Test in a separate container as an alternative
Refer (#2084)
This option puts the end to end testing (pytest-playwright) into a separate Dockerfile and creates a new service called playwright
. This overlooks the current dependency issues that pytest-playwright brings to light when pytest-playwright is added to the CAP Dockerfile.
This strategy allows end-to-end testing to work where a "playwright" container makes api calls and clicks onto a locally running CAP install. The environment isn't reset after each test, which mimicks what would happen in load testing or in normal use -- if reset were needed manual CRUD scripts could be created... but may not be necessary on local/staging...
But this option at the moment allows for CAP tests to be developed while underlying bug fixes are worked on and code is ported.
So the general way pip-tools works is that requirements.in shows the top level python requirements we have (like "our app requires some version of Django"), while requirements.txt shows the specific versions of every dependency and subdependency we're actually tested against (we run against django==x.y.z, and that requires subdependecy foo==a.b.c and so on). Our general approach is to update pinned packages area by area as needed, rather than all at once, because cutting edge versions of python packages often don't play well with each other. To update the pin on a given package you'd use the -P
flag to pip-compile.
The error in your first section is telling you that we have the top level requirements factory-boy
and pytest-playwright
, and in the current pinned versions of everything their subdependencies conflict. But the python community usually sorts those conflicts out -- it's probably just because requirements.txt has a very old version of something pinned. In particular we have an old version of the subdependency faker pinned at the moment: text-unidecode==1.2 (from faker==1.0.2->factory-boy==2.12.0
. (And an old version of factory-boy, though I don't think that's the issue.)
If you tell pip-tools to update the pinned version, which I think is done with fab pip-compile:"-P faker"
, then requirements.txt will update. What I would do to update any dependency is check out a fresh develop
branch, update the dependency, run pip install -r requirements.txt
to install the new requirements, and run fab test
to make sure the tests still work or tweak anything broken by the update. Feel free to bump factory-boy itself to the latest version at the same time, or just faker for now, however far down the rabbit hole you want to go -- hopefully they'll just both update correctly without code changes.
Then once that update is applied, pytest-playwright should go into requirements.in OK.
(The fallback would be to force a subdependency resolution with text-unidecode==x.y
in requirements.in, but that shouldn't be necessary if the relevant packages are up to date.)
I wouldn't mess with updating Fabric -- they chose to change the API drastically in Fabric2, and the python3 port of Fabric1 that's hosted at Fabric3 works fine.
I noticed on Perma requirements.in
at times specifies the version of the dependency for example
however using your suggestion
fab pip-compile:"-P factory-boy"
followed by fab pip-compile:"-P faker" seem to do the trick.
Thnx
Build failing at:
+ docker-compose exec web pytest --cov --cov-config=setup.cfg --cov-report xml -vv
^@Traceback (most recent call last):
File "/usr/local/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 180, in console_main
code = main()
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 136, in main
config = _prepareconfig(args, plugins)
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 314, in _prepareconfig
pluginmanager=pluginmanager, args=args
File "/usr/local/lib/python3.7/site-packages/pluggy/hooks.py", line 286, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 93, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 87, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 203, in _multicall
gen.send(outcome)
File "/usr/local/lib/python3.7/site-packages/_pytest/helpconfig.py", line 99, in pytest_cmdline_parse
config = outcome.get_result() # type: Config
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 80, in get_result
raise ex[1].with_traceback(ex[2])
File "/usr/local/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 932, in pytest_cmdline_parse
self.parse(args)
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1204, in parse
self._preparse(args, addopts=addopts)
File "/usr/local/lib/python3.7/site-packages/_pytest/config/__init__.py", line 1097, in _preparse
self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
plugin = ep.load()
File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 95, in load
module = import_module(match.group('module'))
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "/usr/local/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 170, in exec_module
exec(co, module.__dict__)
File "/usr/local/lib/python3.7/site-packages/pytest_playwright/pytest_playwright.py", line 141, in <module>
pytestconfig: Any, request: pytest.FixtureRequest, folder_or_file_name: str
AttributeError: module 'pytest' has no attribute 'FixtureRequest'
https://github.com/harvard-lil/capstone/runs/7530495223?check_suite_focus=true