Host tests are skipped by default
rth opened this issue · 4 comments
Currently, host tests are skipped by default. For instance in Pyodide,
pytest packages/_tests/test_packages_common.py -k parse_package -v
all tests are skipped.
This is pretty dangerous and should not happen.
In particular, it even passing --rt=host produces another issue in the above command, since it looks like we did changes with this test skipped so it's now broken,
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/_pytest/main.py", line 268, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/_pytest/main.py", line 321, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/_pytest/main.py", line 332, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/_pytest/main.py", line 660, in perform_collect
INTERNALERROR> hook.pytest_collection_modifyitems(
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/home/rth/miniconda3/envs/pyodide-env/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> File "/home/rth/src/pyodide/conftest.py", line 138, in pytest_collection_modifyitems
INTERNALERROR> maybe_skip_test(item, delayed=True)
INTERNALERROR> File "/home/rth/src/pyodide/conftest.py", line 84, in maybe_skip_test
INTERNALERROR> raise AssertionError(
INTERNALERROR> AssertionError: Couldn't parse package name from test_import[host-distlib]. This should not happen!
So the problem is the default runtime is node in which case the host tests are skipped.
I think we should probably remove the "host" runtime altogether and always run host tests. Or there should be another CLI option to skip host tests explicitly. The fact that all regular tests are going to be skipped by installing this package is really surprising to the user. And even for us, it looks like in pyodide we are now skipping a good part of host tests because of this behavior.
cc @ryanking13
I think we should probably remove the "host" runtime altogether and always run host tests. Or there should be another CLI option to skip host tests explicitly. The fact that all regular tests are going to be skipped by installing this package is really surprising to the user. And even for us, it looks like in pyodide we are now skipping a good part of host tests because of this behavior.
Probably we should change the behavior such as:
--rt=chrome: run chrome + host tests--rt=host: run host tests only
+1. but I can also imagine a case where you want to run the chrome tests excluding host tests, and we can probably add an extra flag for that if necessary.
chrome-no-host?