Add an upgrade-all command
pradyunsg opened this issue · 37 comments
it doesnt make sense to add an upgrade_all command until there is a curated package repository
Following the discussion from #6081, I've successfully created a plugin system for pip that seems to work fine with existing packages.
Here is the branch I'm working on (see - plugin.py)
How it works?
- The
search_plugin_scripts()function looks for executables prefixed withPIP_PLUGIN_SCRIPT_PREFIX(currently"pip-plugin-"). So it catches executables likepip-plugin-compile. - It lists them in a 3rd party plugin section at
--help. Auto-complete also works as of now. - When invoking the script, it provides a single JSON encoded string, which helps the plugin determine the environment in which pip is running (see here).
- From there on, the plugin is free to do anything with this information.
This architecture allows for plugins that are independent of the actual package, which I think is quite nice.
It also solves the major recurring problem related to cross-virtualenv execution. Any plugins written with this architecture can be made so that they only need to be installed once on the global python, and then used from within a virtualenv.
I have also added a --disable-plugins switch.
How to actually write plugins?
I've created a small package that will generate a plugin. It allows exposing an existing python CLI as a pip plugin.
Also, Here is a proof of concept that contains some popular packages wrapped as plugins. (including $ pip upgrade!)
It's obviously not ready to be merged. I need to run the tests (and write new ones).
There is no documentation either.
@devxpy Discussion about a plugin API is off topic for this issue.
There's prior discussion at #3999. If you want to make some points about why we should have one, especially ones not already made there, please make them there or in a new issue referencing that.
Sorry about the confusion - I hadn't noticed that in #6081 the discussion had shifted to talk about a plugin API and the MVP was regarding an upgrade command, not a plugin API.
https://github.com/nschloe/pipdate does something like a makeshift global upgrade. Be aware though that upgrading can break things.
https://github.com/nschloe/pipdate does something like a makeshift global upgrade. Be aware though that upgrading can break things.
You're absolutely right, it can break things ! 👍
Here is my attempt of an "almost" robust solution - pipupgrade. Also handles packages that break change.
As this ticket is blocked by the development of the dependency resolver (#988), I thought I would mention here that the team is looking for help from the community to move forward on that subject.
We need to better understand the circumstances under which the new resolver fails, so are asking for pip users with complex dependencies to:
- Try the new resolver (use version 20.1, run
--unstable-feature=resolver) - Break it :P
- File an issue
You can find more information and more detailed instructions here
I think this issue can now progress
Well said, now we need someone to actually progress things. Any volunteers?
@uranusjr I volunteer, any tips on getting started?
Basic workflow might be something like this:
python -m venv venv
venv/bin/python -m pip list --format freeze | \
grep --invert-match "pkg-resources" | \
cut --delimiter "=" --fields 1 | \
xargs pip install --upgradeNote that this installs / upgrades all the packages of the virtual environment in one pip install command. If I understand correctly, pip should therefore resolve all their dependencies and upgrade only to versions without conflicts. An upgrade-all command could thus be a wrapper like this around the pip list and pip install subcommands. Can somebody more knowledgeable about pip please confirm?
I would filter with the --outdated argument in the pip list ;)
And for the concern with the virtual env not updating global packages, probably also with the --local filter?
or would we let this up to the user, who can suply --local and --user and --path ?
Hi, @JensTimmerman. I did not use the --outdated flag on purpose. The pip install command resolves the dependencies only of those packages being (re-)installed or upgraded (at least to my knowledge).
@Bengt got it! we'll leave the filtering up to the user!
However, I'm not seeing the difference in practice, if the packages are not outdated, and not new dependencies of packages that are outdated, they don't need to be considered?
If a package has a dependency that is outdated, the dependency will show up in the list of outdated packages right?
If it's a new dependency, then the package has to be outdated, otherwise there couldn't be a version with a new dependency?
Or am I missing something?
@JensTimmerman I am not sure that I get what you are saying. However, an issue can arise if a current package depends on an outdated dependency: pip list --outdated would list the dependency, but not the dependent package. Thus, pip install --upgrade on only the outdated package would install a version of the dependency package which is incompatible with the dependent package. Only when we run pip install on all the packages, these kinds of issues can be avoided.
@Bengt right, it's for pip to take everything into account for the dependency algorithm.
So Upgrade-all will not upgrade everything if doing so would break an older package.
How will pip handle it when one package depends on a newer version whilst another depends on an older?
How to handle security fixes? Where e.g. I would like to update some encryption related package that just had an update but one of my other packages that uses this can't work with the newer version, and is no longer maintained, I will now never get the security update?
Anyway, I've made a pr with my first attempt at this in #10040 , can you please provide some feedback there?
In this attempt I am letting the current code handle all this, and keeping all options open for the user, so I am not intervening to much in the process.
Seems like pip install -Ur requirements.txt should update all packages, right?
What scenario does the upgrade-all command try to address?
Seems like
pip install -Ur requirements.txtshould update all packages, right?What scenario does the
upgrade-allcommand try to address?
A few use cases which come to mind first for me are when learning or teaching Python, or a module for it, and wanting to experiment with or demonstrate a current version, not an outdated one; when starting a new project and wanting to build and test it using current modules, not outdated ones; or, as I often do, when using Python as a shell-like environment for getting quick answers that are impractical in a standard shell such as Bash.
In my work, I do a great deal more tutoring of college or university students in mostly first-year programming courses than I do Python development, and I'm always wanting to teach what's current. (I always teach through reference to documentation, in order to instill the practice of looking stuff up, which means I'm always learning the latest version as I teach it.)
When not tutoring, I'm often in a Python REPL or notebook, looking for a quick solution to a problem which is impractical by hand, such as algorithmically substituting colours in a large, generated stylesheet to create a dark-mode user style for a large website which lacks one. In other words, I typically use Python more like Inkscape than Xcode.
For those of us who use Python primarily as an application in its own right, not as a means to produce one, pip upgrade-all would serve the same function as brew update && brew upgrade, sudo apt update && sudo apt upgrade, RStudio's Tools menu -> Check for Package Updates…, or so on.
For those of us who use Python primarily as an application in its own right, not as a means to produce one, pip upgrade-all would serve the same function as brew update && brew upgrade, sudo apt update && sudo apt upgrade, RStudio's Tools ...
You really should not use pip to update system dependencies. There may be system package that break on specific versions of packages and break due to the upgrade.
You'll also have issue the next time you run, Eg, apt update, since apt will have no awareness of the change pip made, and overwrite those.
It's really a great way to end up with a broken system.
Makes sense for a virtualenv that's use interactive though 👍🏻
For those of us who use Python primarily as an application in its own right, not as a means to produce one, pip upgrade-all would serve the same function as brew update && brew upgrade, sudo apt update && sudo apt upgrade, RStudio's Tools ...
You really should not use
pipto update system dependencies. There may be system package that break on specific versions of packages and break due to the upgrade.You'll also have issue the next time you run, Eg,
apt update, sinceaptwill have no awareness of the change pip made, and overwrite those.It's really a great way to end up with a broken system.
Makes sense for a virtualenv that's use interactive though 👍🏻
I'll leave it to the originator of this request to discuss why they'd use the proposed update-all, but I don't use pip to update system dependencies. I use Software Update to do that, on my work machine, and it updates /System, /Library, /usr, and so on. Likewise, I use brew to update POSIX dependencies which aren't part of macOS, in /usr/local, wherein the only Python modules are certifi, setuptools, wheel, and pip itself.
Meanwhile, every native macOS application which depends on Python either uses the macOS-provided Python or a Python runtime packaged within the app's own bundle (aside from a one-line AppleScript app I wrote, called "Jupyter QtConsole", and "/Applications/Python 3/IDLE", which is a symlink into /usr/local/…).
I use pip to update ~/Library/Python, of which I am the only consumer. Because of the number of science and GIS majors at my institution, my ~/Library/Python maintains an up-to-date scipy and the like, currently amounting to 80 packages. Updating those packages manually, or with crude hacks like running pip with a command substitution from a string-processed output from another instance of pip, takes time away from studying new features in a module I'll be using and imagining ways to apply those to a topic someone else will be studying.
(But again, I'm just one random who likes the idea, not the one who requested it. And sure, I'll put virtualenv on my list too, after I cover the courses for upcoming semester's new program and the packages relevant for those. It looks interesting.)
Folks, let's keep the discussion about managing system packages separate from this request. That's a different problem related to interaction of lots of tools. It's being tackled separately, and there's likely going to be a PEP and a bunch of process changes in various communities to properly solve that problem.
This feature request is purely a pip-specific feature, about "I have a bunch of packages in an environment, and I want to bump them to the current latest version". That's a different thing from "is pip supposed to manage packages in an environment".
I made a new attempt at implementing a upgrade-all command. in #10491
All feedback is welcome.
Moving the discussion of upgrade-the-world functionality to a separate issue. It doesn't make sense to start this discussion until there's a dependency resolver added.
You shouldn't try to solve the tangle of problems and be smart about making decisions for user to make that run safe in production. Do what asked straight - upgrade everything to the current version. If the env has 100 packages, most likely nothing will get upgraded because few authors of some don't care anymore and require setuptools from stone age. 10 years passed, any results? NOTHING.
You all trying to overthink and complicate a basic feature, and all that cringe fantasy PEP bla bla bla, thats pathetic.
Beta downvotes prove I'm not wrong.
FWIW, my Meta Package Manager project can emulate the missing upgrade-all command for pip with:
$ mpm --pip upgrade --all
Until an --upgrade-all option is officially supported, this did the trick for me:
pip list --outdated | grep -Ev "Package|^-" | awk '{print $1}'| while IFS= read -r line ; do pip install "$line" -U ; doneUntil an
--upgrade-alloption is officially supported, this did the trick for me:pip list --outdated | grep -Ev "Package|^-" | awk '{print $1}'| while IFS= read -r line ; do pip install "$line" -U ; done
I think there are enough shell scripts doing this already. The problem is that this might cause compatibility issues.
But, I wonder if there is already a 'safe' way to do an upgrade all command by now? If we take the output of pip list --not-required(to only get top-level dependencies) and pass it to pip install --upgrade-strategy=eager, this would use pip's dependency resolver? Meaning it would update both top level and transitive dependencies to the latest versions that don't create version conflicts in the current environment?
Folks, let's keep the discussion about managing system packages separate from this request. That's a different problem related to interaction of lots of tools. It's being tackled separately, and there's likely going to be a PEP and a bunch of process changes in various communities to properly solve that problem.
This feature request is purely a pip-specific feature, about "I have a bunch of packages in an environment, and I want to bump them to the current latest version". That's a different thing from "is pip supposed to manage packages in an environment".
Hi, it seems the blocker is resolved, what's the current state of this feature? Are there any plans to implement it?
#10491 seems to have stalled, and no-one else is working on it as far as I am aware.
okay. when can this command be added? it has been 6 years
Feel free to submit a pull request.
Feel free to submit a pull request.
#10491 is open for feedback
Bump.
Linux/macOS with jq:
pip list -o --format json | jq -r '.[].name' | xargs -n 1 pip install -U
Windows with jq:
for /F %i in ('pip list -o --format json ^| jq -r ".[].name"') do pip install -U %i
jq, if not already installed:
https://jqlang.github.io/jq/download/
@utkonos It is better not to use -n 1 with xargs. This allows pip avoid breaking version dependencies among packages being upgraded. Unfortunately this still fails if the dependent package is not being upgraded. To allow this dependency check I think it is better to simply upgrade all the installed packages instead of just old ones:
pip list --format json | jq -r '.[].name' | xargs pip install -Upip will just show notifications for packages which are already at the latest version:
Requirement already satisfied: attrs in ./.venv/lib/python3.10/site-packages (24.2.0)
Code in PowerShell (quickly tested in Windows and Linux):
python -m pip install -U (pip list --format json | ConvertFrom-Json | ForEach-Object { $_.name })pip list -o --format json | jq -r '.[].name' | grep -v urllib3 | xargs -n 1 pip install -U
I have found this one useful as well for when a particular library lags behind or for any other reason needs to be skipped no matter what.
This might not be the purpose of the discussion, but I started to use poetry as package manager and I am enjoying it a lot . Maybe a very google alternative to handle with pip limitations
Yes, higher level environment managers like poetry (and rye, pdm, conda, uv, pip-tools, etc.) are better suited to doing an upgrade all.
They can remove or downgrade transitive dependencies, because they have some understanding of what packages are required or requested by the user, whereas pip only understands what's in the environment now.
Pip, alas, only focuses on installing, it doesn't try to manage the environment itself. Users with those needs should look at those packages (I'm personally following uv as it supports features like constraints that were not as simple in poetry). Or look at writing their own scripts that wraps pip commands.