Wheel incorrectly includes (nested) venvs on incremental build
Closed this issue · 5 comments
Platform: Windows (does not occur on Linux)
Python Version: 3.9-3.11 via Conda
Build Version: 0.10.0
We have an issue (FLAMEGPU/FLAMEGPU2#998) whereby, on Windows when our wheel is constructed via python -m build --wheel, it will build correctly the first time.
However, if the build is repeated (without the directory being cleared), a venv directory is created within the site packages of the first venv directory which is subsequently included inside the wheel (despite not being specified within setup.py).
If a third incremental build is performed, a nested venv directory is created within the site-packages of the second venv. This continues until the file paths become too long and file creation fails killing the build.
This both inflates wheel sizes and causes build failures.
We can work around it by forcing a rebuild and purging the directory, however both the build and venv directories need to be deleted to reset properly (build does contain it's own venv copy inside a subdirectory after the second build).
I'd love to find a way to resolve it properly.
I've copied the first 3 egg-info/SOURCES.txt to gist to demonstrate:
- https://gist.github.com/Robadob/f7bb5a6dc07abc7accf964add2ff6e77 (167 lines)
- https://gist.github.com/Robadob/7f7e9c08104285472b00e861307e4440 (999 lines)
- https://gist.github.com/Robadob/51483e2e574194a613cb1a761726f334 (1831 lines)
If you've got any suggestions for how I might go about resolving or further debugging this it would be great.
We have tried the --no-isolation argument, but it has no impact on this problem.
I'd start by trying to find out what's creating the venv (build isn't) and then by figuring out why setuptools is including it in the wheel (it's really easy to misconfigure setuptools' file inclusion rules). In all likelihood build simply exposed a latent issue.
Thanks I'll look into that tomorrow (I'm just about to leave work).
Unusual that behaviour differs between Windows and Linux.
So I've had a look at the setuptools, appears a pyproject.toml is mandatory, whereas we only appear to have setup.py. I'll try and setup a pyproject.toml, which will hopefully fix it.
Less sure how to track what's creating the venv's, if I had to guess based on my earlier debugging it would be setuptools, I think that's as far as I got dropping print() statements everywhere in the control flow before I got lost (read/write manifest file).
Aha, we create a venv for convenience, that we install the wheel into after build, but this is created inside the working directory containing setup.py. This combined with giving setuptools a very vague way to find stuff for package (find_namespace_packages(), without specifying where). Likely explains it grabbing more than we want.
I've fixed it by moving from flat layout with find_namespace_packages() to src layout with find_namespace_packages(where="src") and package_dir={"": "src"}.
Bit of a mess this fix though, so I'll stash the changes and try find_namespace_packages(exclude=['venv']), as that's a 1 line change rather than a load of CMake path shuffling.
Thanks @layday, your hint got me past where I'd stalled! (I prev looked into this months ago and gave up)
Edit: exclude=['venv'] did not resolve it, so back to plan A, will test that more thoroughly.