Editable installs with paths involving symlinks may include incorrect directories in __path__
vyasr opened this issue · 7 comments
For use cases like those supported by #362 (monorepos containing subdirectories with Python packages) it is not uncommon for the package name directories to be identical to the monorepo directories. If any files are symlinked from outside the Python package source, this causes problems for how scikit-build-core generates the paths used for the redirecting finder when building editable packages. The modules paths that are passed to _make_editable are absolutized, which means that is how they are embedded into the install call in the _${pkg}_editable.py file. The end result is that the __path__ produced when the package is imported may erroneously contain both the Python package directory and the higher-level directory in the monorepo.
I've put together an example in #648. Happy to work on a solution with a bit of guidance on preferred approaches, but also maintainers should feel free to push to my branch if there's an easy solution that you'd like to see. Thanks in advance either way!
We don't support subdirectories all that well currently either, so it's something I'd also like to see improved. We might need a setting to tell scikit-build-core that it's N levels deep in a repo. Symlinks are a bit tricky, as they can't be passed through a zip file, either for wheels or editable wheels (due to a old CPython bug) - I'm guessing this means they will need special handing? Do you have any ideas currently?
For the most cases if we are using editable installs, we have a local git repo where the sym links are already setup, so the zip issue shouldn't be that important.
For the most cases if we are using editable installs, we have a local git
The problem is that editable installs in PEP 660 are set up as a transient wheel, which means they pass through a zip file, even if the zip file is not kept around. This only affects things that are "installed", though.
Sorry I lost this notification in the scramble. Yeah, the fact that we pass through a zip file is an issue I wasn't sure how to overcome completely either. I think it would be a reasonable for such files to not be truly editable for an editable install but to instead be copied over. That would be a slightly unfaithful representation of the editable install, but it would avoid what is probably a worse problem in setting attributes of the installed package (the __path__) incorrectly. What do you think?
That would be fine, I think. We could also add more modes to choose from, https://tttapa.github.io/py-build-cmake/Editable-install.html has some interesting ideas. There is a "symlink" idea using a .pth file and a hidden folder in the source.
Hmm yeah that solution does seem like a reasonable out. The "hook" approach is basically scikit-build-core's "redirect". The "symlink" approach could be added analogously. Then you would presumably categorize files by type to determine whether they are symlinked or copied in.