setup.py leaves build, dist, .egg-info etc + even clean doesn't remove them
stuaxo opened this issue · 17 comments
By default, setup.py leaves build, dist, ${project_name}.egg-info directories in a projects directory.
I'm not convinced these need to be here these days, certainly feels messy to have these.
Putting all this output into one directory would be some improvement. But maybe there's a better solution ?
python setup.py clean doesn't even delete these without adding hacks.
I end up putting this in every setup.py
class CleanCommand(Command):
"""Custom clean command to tidy up the project root."""
CLEAN_FILES = './build ./dist ./*.pyc ./*.tgz ./*.egg-info'.split(' ')
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
global here
for path_spec in self.CLEAN_FILES:
# Make paths absolute and relative to this path
abs_paths = glob.glob(os.path.normpath(os.path.join(here, path_spec)))
for path in [str(p) for p in abs_paths]:
if not path.startswith(here):
# Die if path in CLEAN_FILES is absolute + outside this directory
raise ValueError("%s is not a path inside %s" % (path, here))
print('removing %s' % os.path.relpath(path))
shutil.rmtree(path)Other solutions I've found which have been refined a stage above "here's what's worked for me" code snippets here and on stack overflow:
-
https://github.com/dave-shawley/setupext-janitor - looks to be clean, well featured and documented. Hasn't been touched in awhile and there's a couple easy win PR's open. Might not take much to dust off and press into service though.
-
https://scikit-build.readthedocs.io/en/latest/_modules/skbuild/command/clean.html - part of a big community so should be more tested. Has the advantage being easy to read and understand the essential bit. Looks harder to drop into any old project though.
I've made good headway with setupext-janitor starting from @ionelmc's fork https://github.com/ionelmc/setupext-janitor which fixes namespace. There's a blocking issue though on Windows, ... clean --dist exits early with "error: don't know how to create RPM distributions on platform nt" (dave-shawley/setupext-janitor#12). I'll post progress there as I can, label:help-wanted though as I'm out of my depth. ;-)
It might be worth opening a PR marked WIP so we can easily see the changes.
@stuaxo Sorry, I don't follow. A PR here in setuptools?
I've identified where the error occurs in setuptools now, but don't know how to proceed. dave-shawley/setupext-janitor#12 (comment)
Sorry for the noise, was replying on my phone in a hurry :)
I've got setupext-janitor working and issued a PR to the base master dave-shawley/setupext-janitor#14
It would be very nice if the behaviour of clean in setuptools could be improved; the current behaviour is really annoying IMHO. @maphew's updated version of setupext-janitor works nicely (thanks!). However, the original owner/maintainer (@dave-shawley) doesn't seem to be much interested any more (?), unfortunately.
Is there a chance to get the patched version released on Pypi somehow? Or even be integrated into setuptools properly?
Rather than trying to clean up files from the source directory, would it be better to be able to do out-of-source package builds? Something like this: cd /tmp/build && python /path/to/setup.py. That way those generated files never touch the source directory in the first place.
@cmarquardt these and other recent comments have woken up setupext-janitor and a recent version has been published to pypi. @dave-shawley wasn't receiving notifications and didn't know there was any interest or significant downstream use.
@gary-jackson are you suggesting this be something setup.py does automatically or as documenting best practice instructions?
Not speaking for @gary-jackson but I'd like it to do this automatically, not sure if it will break someones build process though ? OTOH not leaving random files around has to be a pretty good benefit.
@maphew I suggest that it be added as functionality (it doesn't work that way currently), then documented as a best practice. It shouldn't break the way it's done now.
It would be very nice if the behaviour of clean in setuptools could be improved; the current behaviour is really annoying IMHO.
I completely agree. I find it very strange that there is no option to select a different output folder AND that the clean command does not really remove the generated artifacts.
I think a out-of-source build and proper clean should be natively supported by setuptools, as per previous comments.
Trying to make a nice workaround for this stuff, I couldn't manage with changing workdir - it fails with line "error: package directory 'my_root_package' does not exist".
In my case I just build a source distribution with no data files, extensions etc, just plain python archive. I found a workaround for this issue - having workdir at repo root as usual, use command:
python setup.py egg_info --egg-base tmp/build sdist --dist-dir tmp/dist
How it works:
- sdist anyway runs egg_info command, all commands run only once, so you can freely add all the commands you already use and override their options.
- egg_info allows for changing the path where .egg-info directory will be written, so this way I get rid of .egg-info files in source root.
- --dist-dir thing changes the destination for output archive.
- All others artifacts are cleaned up by default.
- If you have more complex builds let's say with extensions, most probably you should also be able to override their options by this method.
Not really convenient solution, but it does not require any extra code.
I had problems with that, I added data files to MANIFEST.in, added include_package_data=True to setup() and built the package again (python setup.py sdist) but setuptools won't install the data files until I manually clean /dist and /*.egg-info dirs, past builds were interfering in the behavior of new builds, if that's allowed to happen then setuptools should also clean that directories along with /build (python setup.py clean --all should clean /dist and /*.egg-info too).
If you find an easy-to-use built-in solution, it would be great to post it here as well: https://stackoverflow.com/questions/64952572/output-directories-for-python-setup-py-sdist-bdist-wheel
Thanks!
Like others who've commented here, I have had to resort to a custom clean command to remove the dist directory.
It would be good if this could be fixed in setuptools. A user would reasonably expect that python setup.py clean --all would clean all directories. However, I haven't come across a situation where I want a partial clean so I'm not sure there is much point in a --all flag: clean should just clean everything.
class CleanCommand(clean):
"""
Custom implementation of ``clean`` setuptools command."""
def run(self):
"""After calling the super class implementation, this function removes
the dist directory if it exists."""
self.all = True # --all by default when cleaning
super().run()
if exists("dist"):
log.info("removing 'dist' (and everything under it)")
rmtree("dist")
else:
log.warn("'dist' does not exist -- can't clean it")