COVERAGE_CORE=sysmon with branch coverage can be 2x slower than default in some cases
lesteve opened this issue · 5 comments
Describe the bug
Using COVERAGE_CORE=sysmon with branch coverage can be 2x slower than default. Note the 2x likely depends on the code whose coverage is measured. Trying different variations of the Python snippet below I have seen vary from ~20% to 2x degradation roughly.
To Reproduce
Python script:
# test-coverage.py
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TunedThresholdClassifierCV
X, y = load_breast_cancer(return_X_y=True)
for i in range(10):
model = TunedThresholdClassifierCV(
estimator=LogisticRegression()
).fit(X, y)Script to run:
# Just for completeness you need to have Python 3.12 to be able to use sysmon
pip install scikit-learn coverage
git clone https://github.com/scikit-learn/scikit-learn --depth 1
# Note: there are some scikit-learn specific warnings, so the grep is to keep only the timing info
(time coverage run --branch --source sklearn /tmp/test-coverage.py 2>&1) | grep total
# on my machine: 9.75s user 0.09s system 129% cpu 7.616 total
(time COVERAGE_CORE=sysmon coverage run --branch --source sklearn /tmp/test-coverage.py 2>&1) | grep total
# on my machine: 16.88s user 0.10s system 114% cpu 14.798 totalI read the other performance issues with coverage and Python 3.12 for example #1665 and python/cpython#107674 and this seems like a slightly different issue. This may be related to the fact that only statement coverage is using sysmon and not branch coverage according to #1746 (comment) but I still find the performance degradation somewhat surprising.
How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
- What version of Python are you using? 3.12.4
- What version of coverage.py shows the problem? The output of
coverage debug sysis helpful.
-- sys -------------------------------------------------------
coverage_version: 7.5.4
coverage_module: /home/lesteve/micromamba/envs/test-coverage/lib/python3.12/site-packages/coverage/__init__.py
core: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: /home/lesteve/dev/cpython/.coveragerc
/home/lesteve/dev/cpython/setup.cfg
/home/lesteve/dev/cpython/tox.ini
/home/lesteve/dev/cpython/pyproject.toml
configs_read: -none-
config_file: None
config_contents: -none-
data_file: -none-
python: 3.12.4 | packaged by conda-forge | (main, Jun 17 2024, 10:23:07) [GCC 12.3.0]
platform: Linux-6.9.8-arch1-1-x86_64-with-glibc2.39
implementation: CPython
gil_enabled: True
executable: /home/lesteve/micromamba/envs/test-coverage/bin/python3.12
def_encoding: utf-8
fs_encoding: utf-8
pid: 510639
cwd: /home/lesteve/dev/cpython
path: /home/lesteve/micromamba/envs/test-coverage/bin
/home/lesteve/micromamba/envs/test-coverage/lib/python312.zip
/home/lesteve/micromamba/envs/test-coverage/lib/python3.12
/home/lesteve/micromamba/envs/test-coverage/lib/python3.12/lib-dynload
/home/lesteve/micromamba/envs/test-coverage/lib/python3.12/site-packages
environment: CONDA_PYTHON_EXE = /home/lesteve/micromamba/bin/python
HOME = /home/lesteve
PYTHONPATH =
command_line: /home/lesteve/micromamba/envs/test-coverage/bin/coverage debug sys
sqlite3_sqlite_version: 3.46.0
sqlite3_temp_store: 0
sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-12.3.0, DEFAULT_AUTOVACUUM,
DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
DEFAULT_WORKER_THREADS=0, DIRECT_OVERFLOW_READ, ENABLE_COLUMN_METADATA,
ENABLE_DBSTAT_VTAB, ENABLE_FTS3, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4,
ENABLE_FTS5, ENABLE_GEOPOLY, ENABLE_MATH_FUNCTIONS, ENABLE_RTREE,
ENABLE_UNLOCK_NOTIFY, MALLOC_SOFT_LIMIT=1024, MAX_ATTACHED=10,
MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500, MAX_DEFAULT_PAGE_SIZE=8192,
MAX_EXPR_DEPTH=10000, MAX_FUNCTION_ARG=127, MAX_LENGTH=1000000000,
MAX_LIKE_PATTERN_LENGTH=50000, MAX_MMAP_SIZE=0x7fff0000,
MAX_PAGE_COUNT=0xfffffffe, MAX_PAGE_SIZE=65536, MAX_SQL_LENGTH=1000000000,
MAX_TRIGGER_DEPTH=1000, MAX_VARIABLE_NUMBER=250000, MAX_VDBE_OP=250000000,
MAX_WORKER_THREADS=8, MUTEX_PTHREADS, SECURE_DELETE, SYSTEM_MALLOC,
TEMP_STORE=1, THREADSAFE=1
- What versions of what packages do you have installed? The output of
pip freezeis helpful.
coverage==7.5.4
joblib==1.4.2
numpy==2.0.0
scikit-learn==1.5.1
scipy==1.14.0
setuptools==70.2.0
threadpoolctl==3.5.0
wheel==0.43.0
Expected behavior
Usins COVERAGE_CORE=sysmon improves performance and in the worst case may not improve it much but at least does not degrade it
Additional context
We originally saw this in the scikit-learn CI scikit-learn/scikit-learn#29444 (comment).
Yes, the sysmon support is not yet in place to do a good job with branch coverage. I could add a warning to that effect if it would help.
Thanks for your answer, I guess this was more of a "is this expected" question than a call for action.
I am not sure whether a warning would be that useful, in particular because in our use case we use coverage via pytest-cov and I think pytest would capture the warning which would kind of hide it amongst a number of other warnings that happened when we run our tests.
Another option: refuse to run with COVERAGE_CORE=sysmon and branch=True.
Why not error indeed. I guess it would make sense to do this if you think that in a vast majority of cases the performance will be worse using COVERAGE_CORE=sysmon with branch=True compared to branch=True with COVERAGE_CORE unset.
In my case, we did see a (moderate) improvement in performance when setting COVERAGE_CORE=sysmon while still using branch coverage. So I at least like that combining them does not result in an error (the performance improvement was much bigger if I also disabled branch coverage, but for now we prefer retaining that even with some perf cost).