Deprecate call to `setup.py install` when `wheel` is absent for source distributions without pyproject.toml
sbidoul opened this issue · 50 comments
This is one of a series of deprecations meant to collect feedback on the ultimate goal of always installing source distributions by first building a wheel and then installing from it.
If you're seeing this error, it's a symptom of either:
-
A package failing to build. Scroll up in the output to see if "Building wheel" failed for the same package. If yes, then you want to investigate that issue and not this warning.
-
Or,
wheelnot being installed in your environment.If you're a user, install that package in the environment to avoid hitting this issue.
If you're the package author, make sure you can install the package with
--use-pep517passed into pip.
This specific issue is about the case when pip has to call setup.py bdist_wheel because the wheel package is not installed.
In that case, pip currently displays deprecation warning and continues installation by attempting a setup.py install.
In a future version, pip will either (to be decided):
fail right awayattemptsetup.py bdist_wheeland let that fail (the error message might be unclear in that case)- attempt an isolated build in PEP 517 mode with the default build system
The recommended way to silence this warning is to enable the --use-pep517 flag, which will in turn install setuptools and wheel in an isolated environment where it will attempt to build a wheel and install it.
Towards #8102
In #11456 we are targeting the removal of this setup.py install fallback to pip 23.1 (April 2023).
To be decided is if we do an isolated or non isolated pep 517 build as a replacement.
To be decided is if we do an isolated or non isolated pep 517 build as a replacement.
Hmm... Isn't this a fallback after the legacy setup.py-based build of a wheel fails -- in which case, we could basically remove this logic with no additional replacement; surfacing the initial build error directly?
NVM, I misunderstood what you've said here. :)
Hello, might be experiencing an issue related to this. I'm trying to install a pip package (yfinance) that relies on lxml, and I'm receiving this error:
When attaching --use-pep517, the error becomes:
Had a hard time following along the messages above, so if I missed a solution, please let me know
Hey @ccmilne
You can run pip install <package-name> --use-pep517 as a fix for now
@ccmilne the cause of the error is not in the screenshots you posted. It's likely displayed a few lines above the part you posted.
It's most likely a compilation issue with lxml (due to a missing build dependency), and probably not related to this issue, though.
Resolved. Rolled back to Python 3.9.0 (from 3.10) and everything disappeared. Thanks!
I just got an issue from one of the users of my package peewee. I'm a little confused about how to proceed as Peewee makes heavy use of the dynamic nature of setup.py to perform the build differently depending on whats available in the users python environment. Specifically,
- When cython is available we will cythonize the optional extension files
- When sqlite headers are available we will compile the optional extensions
- If sqlite headers are not available we do an installation without the optional extensions.
Not sure how to cram this into a "toml" file.
I think the issue for this user is they have a very new pip but are missing the wheel package (which, by the way, isn't even at a 1.0 release!). My library is not "legacy" code - it uses standard library stuff (which wheel is also not a part of). I hope you will reconsider deprecating the setup.py install fallback.
@coleifer If you just add a pyproject.toml file to your project containing
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"
that will probably work fine. The new approach doesn't stop you using dynamic builds in setup.py, all you need in pyproject.toml is enough to say "I'm using setuptools (with wheel for building wheels)" and the rest is probably the same as it always was.
The one thing about your description above - "When cython is available". Under the pyproject.toml based build process, the build doesn't run in the user's runtime environment. So the user doesn't need build tools like cython installed. But conversely, if they have them installed, the build won't use them. You can add "cython" to the "requires" item in pyproject.toml, and then pip will automatically install cython for all your users (so they will get the optional extension files cythonized). If you don't want that, and prefer your users' builds to run in their default environment, you should tell them to use --no-build-isolation, but in that case (as now) they will be responsible for installing setuptools and wheel before doing the build.
Hope this helps.
Thanks for the very speedy and thorough response @pfmoore - yes for the moment I've just added a pyproject file:
[build-system]
requires = ["setuptools", "wheel"]
I'm not sure I understand the difference between yours and the stated fallback "setuptools.build_meta:__legacy__", but in any case I will keep an eye on that particular setting. It's too bad about needing to specify a flag for the cython stuff, but I don't want to make cython a dependency as probably 98% of peewee users don't touch the optional extensions. It may be the case going forwards that it's easier to split those off into their own package, I don't know and not your problem in any case.
RIP to setup.py install. I'm sympathetic to the desire to simplify and streamline things, but I'm frustrated that it is being deprecated.
Hi there,
I'm the maintainer of OctoPrint, which has a plugin system. Plugins are python packages that get installed through pip (usually). In order to ensure that plugins aren't accidentally installed into the wrong environment where OctoPrint isn't present, plugins check for OctoPrint's package in setup.py and refuse an install if it isn't there. Which - in an isolated build environment - will be the case.
I'm currently trying to figure out the best way to support our plugin authors in preparing their plugins for a change in pip's behaviour in the future when it comes to installing from source tarballs on the end-users hardware (most common), and right now I'm utterly confused as to what will and won't be supported come April 2023's pip release.
Specifically: Will pip install http://example.com/some_source_tarball.zip run the build in an isolated or not isolated environment, if wheel is present in the current (virtual) environment?
My confusion primarily stems from the sentence in the first post of this issue:
This specific issue is about the case when pip has to call setup.py bdist_wheel because the
wheelpackage is not installed. In that case, pip currently displays deprecation warning and continues installation by attempting a setup.py install.In a future version, pip will:
* attempt an isolated build in PEP 517 mode with the default build system
Does that mean, this isolated build will only be attempted if wheel is not installed and otherwise everything just works like it does today, the build will be done by running setup.py in the current (virtual) environment? Or does it mean that this isolated build will be the default for setup.py based installs.
Some clarification would be very helpful here. And yes, I know that pyproject.toml is the future, but we are looking at hundreds of plugins, often with an unknown maintenance state, inexperienced Python developers at their helm, and with most of them still not even fully moved over to Python 3, so I hope you understand that "Just move to pyproject.toml" is sadly not the solution for this community at all.
Best regards,
Gina
Does that mean, this isolated build will only be attempted if wheel is not installed and otherwise everything just works like it does today, the build will be done by running setup.py in the current (virtual) environment?
Yes.
@coleifer But the comment specifically was not talking about the pyproject.toml part of things. They are using non-pyproject.toml source builds, and the sentence is correct for those packages.
But the comment specifically was not talking about the pyproject.toml part of things. They are using non-pyproject.toml source builds, and the sentence is correct for those packages.
Right, but isn't the whole underlying issue that projects not using pyproject.toml will not be installable in the future without the special command-line flag?
“In the future” being the key here. I’m assuming the conversation here is about deprecating setup.py install when wheel is absent. If the topic is to change to enforcing build isolation globally, I would suggest opening a new issue.
Can we refocus the discussion here? This issue is very specifically about the case where
- The user tries to install a package, which does not have a
pyproject.tomlfile. - The user does not have the
wheelpackage installed in their runtime environment.
In that case, pip currently reports that it would like to build a wheel using setup.py bdist_wheel but the lack of the wheel package prevents this, so it will fall back to setup.py install.
What we're doing here is adding an additional warning saying that in April next year, we'll switch to a different fallback, namely to set up an isolated build environment that does contain wheel, and build the project using that environment.
So the only change right now will be to add a new warning. And in April next year, if people don't want the new behaviour, they can ensure that wheel is installed in the environment before installing their project, and it'll continue working just the same as it does right now if wheel is installed.
Yes, there are longer term plans to switch to build isolation by default for all projects. But even then, there's still going to be a --no-build-isolation flag to request a non-isolated build. So if you can't work with isolated builds, you don't have to. It's just that you will need to be explicit about that.
We do want feedback from projects that will be impacted by these changes. That's the whole point of the extended and phased deprecation process, after all. But over-reacting and assuming the worst really won't help us get a clear picture of what we need to do to help projects with the transition.
@coleifer Just to be absolutely clear, my reply to you was specifically about your question regarding how to start using pyproject.toml. Having just checked your project, I assume this was triggered by coleifer/peewee#2647? (And by the way, having seen your comment on that issue, I'm going to finish this message and then I won't comment further on your case - I'm not willing to help out any more if that's your attitude 🙁) You could have responded to that by simply asking the user to install wheel for now, while you consider your further options. And there's no immediate need for you to do anything more than that. When we do remove the setup.py install path, just tell your user to specify --no-build-isolation as well as installing wheel. That would cover you for a good length of time - certainly long enough for you to look into how to handle pyproject.toml without needing to panic.
What we're doing here is adding an additional warning saying that in April next year, we'll switch to a different fallback, namely to set up an isolated build environment that does contain wheel, and build the project using that environment.
Gotcha - yes, I was under the impression from reading the initial bug report that the project would somehow not be installable:
DEPRECATION: peewee is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at #8559
The line "pip 23.1 will enforce this behaviour change" is what alerted one of my users and prompted them to open an issue. I was concerned because I read that as saying "your package won't be installable without making some changes". I don't know pip's release schedule so I wasn't sure how much time until 23.1 is released, nor was it clear to me initiially what the proper fix was for my project given its non-standard use of setup.py.
I wrote in my initial response to my user:
At any rate, I'm going to hold off on any knee-jerk changes to this library and suggest for now that you install wheel first.
After receiving your initial answer, the nature of the issue and fix was more clear to me, and is further clarified by your latest comment that there will continue to be an additional fallback. I really appreciate your speedy and thorough responses @pfmoore. I know it takes time to sift through a users comments on an issue, and mental effort to translate their problem into a correct path forward.
We should maybe tweak this specific deprecation message to not mention pyproject.toml, and just suggest installing the wheel package or enabling --use-pep517.
While correct, the current message seems to mislead users into creating issues to request the addition of a pyproject.toml in projects where in most cases it will not be necessary.
On the other hand, raising awareness about pyproject.toml is also desirable, so... not sure.
We should maybe tweak this specific deprecation message to not mention pyproject.toml, and just suggest installing the wheel package or enabling --use-pep517.
That's what #8560 originally did. Looks like what finally got implemented changed this 🙁
On the other hand, raising awareness about pyproject.toml is also desirable, so... not sure.
I'm also not sure. I think projects should be aware of the ultimate goal, the reasons behind it, and the implications for them. If they aren't, then we'll get people saying we're repeatedly changing things throughout the deprecations (because they don't see the bigger picture) and so everyone will get frustrated.
On the other hand, there's only so much context we can put in a warning message...
Do we have an overall issue for the removal of the setup.py code path and its implications? If so, maybe the message could say "See issue #xxxx for the background".
Will pip install http://example.com/some_source_tarball.zip run the build in an isolated or not isolated environment, if wheel is present in the current (virtual) environment?
It'll behave the same as today, if wheel is installed -- i.e. without isolation if there's no pyproject.toml. That will not change due to this issue, but as @uranusjr alluded to, it will be changing with a similar message in a future pip release.
Is it necessary to include wheel in the pyproject file? I got a comment this morning stating:
Remove the redundant wheel dependency, as it is added by the backend automatically. Listing it explicitly in the documentation was a historical mistake and has been fixed since, see: pypa/setuptools@f7d30a9
However in this thread and on the pyproject doc, wheel is recommended to be added.
If memory serves, it's necessary to list wheel if you want your package to be buildable with pip versions older than 19.0. Otherwise, you don't need it.
Is it necessary to include wheel in the pyproject file?
No, setuptools will specify wheel as a dynamic build dependency if you don’t specify it in build-system.requires.
For those that find it easier to understand things by reading the code.
problem dpkg how to fix?
What do I put in the terminal to actually do this? I'm trying to install pandasgui and I'm using this and it's still not installing:
pip install pandasgui --use-pep517
I get this message:
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for wordcloud
Failed to build wordcloud
ERROR: Could not build wheels for wordcloud, which is required to install pyproject.toml-based projects
@astephan023 The real error should be in your console a little bit above the one you pasted, and it is likely something you'll need to raise with the wordcloud project.
im having a error with the dlib download
Hello,
Where and how to enable the --use-pep517 flag?
ok no the building wheels issue is legit that astephan023 posted. I am getting the same error while installing grpcio, typedb client etc multiple files. Idk what the problem is the error is Building wheel for grpcio (pyproject.toml) did not run successfully. and then the last message is the same as what astephan says
--use-pep517 doen't work, pls help
- You didn't provide the full error output.
- If you had, I'd confidently expect that it says this is not an issue with pip, but with MySQL-python, and you'll need to raise the problem with the authors of that package.
- You didn't provide the full error output.
- If you had, I'd confidently expect that it says this is not an issue with pip, but with MySQL-python, and you'll need to raise the problem with the authors of that package.
i decided use postgre instead :')
Tried different ways to solve it but still popping.
You can install the wheel package in that environment before installing other packages.
You can install the
wheelpackage in that environment before installing other packages.
Sir how to get emojis_list
$ pip install gitpython datetime emoji
import random
import emoji
from datetime import datetime, timedelta
from git import Repo
# File path for the data file
FILE_PATH = './README.md'
# Initialize a Repo object for the current directory
repo = Repo('./')
# Make only 1 commit to the current branch
for i in range(1):
# Generate a random date within the past year
x = random.randint(0, 54)
y = random.randint(0, 6)
date = (datetime.now() - timedelta(days=365)) + \
timedelta(days=1 + x * 7 + y)
# Get a list of all available emojis
emojis_list = emoji.emoji_list('')
# Select a random emoji from the list, or handle the exception if the list is empty
try:
emoji = random.choice(emojis_list)
except IndexError:
print("The list is empty, cannot choose a random element.")
continue
# Read the contents of the README.md file
with open(FILE_PATH, 'r') as f:
lines = f.readlines()
# Modify the first line
lines[0] = '# {} Random Commit Generated on: {}\n'.format(emoji, date.strftime("%d/%m/%y"))
# Write the updated content to the file
with open(FILE_PATH, 'w') as f:
f.write(''.join(lines))
# Generate a commit message with the random emoji
commit_message = '{} Random Commit Generated on: {}'.format(emoji, date.strftime("%d/%m/%y"))
# Add the file to the staging area and commit it
repo.git.add([FILE_PATH])
repo.git.commit('-s', '-m', commit_message, '--date', date.strftime('%Y-%m-%d %H:%M:%S'))
# Push the commits to the remote repository
repo.git.push()The list is empty, cannot choose a random element.
so should we just always use --use-pep517 flag?
No, here's what your options are:
- installing
wheelin the environment where you run pip. This means that pip will be able to build wheels usingsetup.py bdist_wheel. - uninstalling
setuptoolsin the environment where you run pip. This basically means that pip will use the newer build system by default. - pass
--use-pep517to pip, if you're using it as a part of in those environments will suppress the message for end users.
Notably, python -m venv doesn't install wheel by default and... well... we want the default behaviour to change in those environments from using setup.py install to building a wheel and installing from that (hence the deprecation warning, before making the change). :)
py -m pip install -r requirements.txt --use-pep517
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
py -m pip install -r requirements.txt --use-pep517 ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'
@AtilaPeixoto do you have a requirements.txt file?
Hi @AtilaPeixoto! This isn't the right place for your issue. I think that Python Discord's #help-* channels, python-list or python-tutor mailing lists,or StackOverflow would be better channels to get support for the issue you're facing.
You’re asking pip to install from a file that doesn’t exist. You’ll want to try and understand why you’re invoking pip that way and if that’s something you want to do.
Just to confirm, attempt an isolated build in PEP 517 mode with the default build system means that packages without .whl files such as https://pypi.org/project/parse/#files will still install with pip after the 23.1 update? I maintain a repo, https://github.com/seleniumbase/SeleniumBase, which has a dependency on https://github.com/behave/behave, which has a dependency on parse, which currently has no .whl file in PyPI. Rather than wait around to see if things fail later when pip 23.1 is released, I'd like to take action sooner if necessary.
that packages without .whl files such as https://pypi.org/project/parse/#files will still install with pip after the 23.1 update?
@mdmintz in your case, yes, as in very many similar cases. You can confirm it with
pip install --use-pep517 behave,- or by installing
wheeland runningpip install behave.
The packages that will fail are those which don't have a pyproject.toml AND for which setup.py bdist_wheel does not work for some reason.
what can be done with this?
DEPRECATION: bs4 is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at #8559
WARNING: The script behave.exe is installed in 'C:\Users\me\AppData\Roaming\Python\Python310\Scripts' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
All can run test using:
- sudo su
- python3 -m venv myenv
- source myenv/bin/activate
- pip install -r requirements.txt
DEPRECATION: lru-dict is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option.
@ShawnXpro Seems that "pip install -U --use-pep517 "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-macosx_10_15_universal2.whl"" does not want to build gprcio once again.. any idea?
Locking to prevent this from being flooded with "I need help, tell me what to do" queries. Please see #8559 (comment) and #8559 (comment). I've also edited the issue description to direct users to what they can do if they see this warning.