Add "upgrade" and "upgrade-all" commands
vbabiy opened this issue · 250 comments
(Updated to reflect reality in April 2020)
This issue was initially scoped for changing the behavior of pip install --upgrade, by adding a new upgrade command with a different behavior when upgrading packages. The older recursive "eager" default caused grief for many (#304) However, after a lot of discussion on this, the approach taken was to add the --upgrade-strategy flag. The flag initially had a default of "eager" which was changed to a better default in #4500.
The tracking issue for "upgrade all packages" functionality is #4551.
"upgrade" is a trivial alias for "install --upgrade". Need to think a bit more
about "upgrade-all"; for one thing, I presume it would only upgrade packages
inside sys.prefix? (i.e. if you're inside a virtualenv, it wouldn't try to
upgrade global packages). That would be a reason to move
UninstallPathSet.can_uninstall() to a more generically-named function (or
method of InstallRequirement), so it provides generic "can I touch this?"
decisions.
Original Comment By: Carl Meyer
For the record, I think that seems like good idea, given the ability to
uninstall before upgrading. Although I'd prefer an --all option for
upgrade instead of an own upgrade-all command.
For the matter of can_uninstall(), I agree.. this is probably handy to have
globally anyway.
Original Comment By: Jannis Leidel
I'm not entirely unopposed to upgrade as an alias for install --upgrade. But
it seems a bit trivial.
upgrade-all requires you to figure out what is "upgradable". Probably one
prerequesite is that it lives in <sys.prefix>/lib/pythonX.Y/site-packages
(simply under sys.prefix isn't enough).
If we allow something like "zip import" (to bring a package from the parent
environment into a virtualenv environment) then probably packages in that
parent environment shouldn't be upgraded, but it's not 100% clear that is what
the user will expect.
I tried uninstalling an editable package with "pip uninstall" and it quite
reasonably offered to remove the .egg-link and update easy-install.pth. But it
couldn't have reasonably upgraded the package, so can_uninstall is somewhat
different from can_upgrade.
Original Comment By: Ian Bicking
Yeah, you're right that can_uninstall and can_upgrade are different.
I would think if we had "pip import" we still wouldn't want to upgrade
imported global packages; but (along with editables) it might be worth a "not
upgrading this" warning to the console.
Original Comment By: Carl Meyer
+1 for this bug
Original Comment By: smyrman
(echo pip; pip freeze | awk 'BEGIN{FS="=="}{print $1}') | xargs sudo pip
install -U
This should upgrade upgrade all installed packages (including pip itself). If
you run it in virtualenv you probably don't need to use sudo.
Of course it has high risk of failure -- if upgrading one of the packages
fails the whole process will fail (it's similar to port upgrade outdated in
MacPorts).
Original Comment By: Tomasz Elendt
+1 for upgrade --all
Why at the moment all Python module management facilities have to suck? Why no
one provides simple upgrade + upgrade --all command?
Original Comment By: Anonymous
I wouldn't mind taking a shot at an implementation, but first a few questions.
Is the general consensus that a new "upgrade" command that supports a '--all'
option be added to pip?
Running pip upgrade should only affect the environment it is running in. If
run from a virtualenv then only packages local to that env will be upgraded;
same for non-virtualenv's
Original Comment By: Kelsey Hightower
Kelsey: from my reading of the above discussion, I don't see any real
opposition to it. I think it's a fine addition to make. The main edge case is
editable VCS installs - as Ian suggested, I think an "upgrade" command
shouldn't touch those. Defining what "upgrade" means in the context of all the
editable possibilities (including local repos installed editable that have no
origin) would be next to impossible, I think, and even if some halfway-working
definition could be put together, it would only increase the maintenance
burden of the already-fragile VCS backends. But for non-editables -- go for
it!
Original Comment By: Carl Meyer
Carl: Cool, I will get started and update this ticket with the results.
Original Comment By: Kelsey Hightower
While working on the upgrade command the following questions came up:
- What methods should pip upgrade support to specify which packages to
upgrade? Should we support a requirements file? - How should pip upgrade handle packages that are not already installed?
Should we install missing packages?
pip upgrade use cases and how to handle them:
# pip upgrade some_installed_package
Try and locate a package that satisfies the requirement. If the
requirement is not satisfied upgrade to the requested version. This includes
upgrading to an older version.
# pip upgrade --all
Locate all installed packages (non-editables) and update them to a new
version if available.
# pip upgrade some_other_package
Warning: some_other_package not installed, use pip install
some_other_package.
My goals are to keep the upgrade command really simple. You can upgrade
specific non-editable packages to a new or older version; or you can upgrade
all non-editable packages to a newer version.
Thoughts?
Original Comment By: Kelsey Hightower
I think "pip upgrade" should be an exact alias for "pip install --upgrade" as
it works now. That implies that yes, it installs requested packages if they
aren't installed, and yes, it accepts requirements files with -r. His should
be doable with almost no new code.
Upgrade --all will require some code for finding the list of currently
installed upgradable packages; then it should just pass that list to install
--upgrade, as above.
Original Comment By: Carl Meyer
Carl, thanks for the reply. I have pretty much taken the path you have
described. Later today I should be able to post some example runs.
Original Comment By: Kelsey Hightower
Got most of the code done, just a little polish left before I post the code
for review.
TODO:
- Tests
- Filter requirements file; add non-editables to list of packages to
upgrade.
Running pip using the upgrade command
# pip upgrade --all
All packages up-to-date
# pip upgrade --all -v
Packages installed at latest version:
pip: 0.8.2 (latest)
distribute: 0.6.14 (latest)
Python: 2.7.1 (latest)
wsgiref: 0.1.2 (latest)
All packages up-to-date
# pip upgrade PyYAML
Package updates available:
PyYAML: N/A (installed) 3.09 (latest)
Downloading/unpacking PyYAML
Downloading PyYAML-3.09.tar.gz (238Kb): 238Kb downloaded
....
Successfully installed PyYAML
Cleaning up...
# pip upgrade --all -v
Packages installed at latest version:
pip: 0.8.2 (latest)
distribute: 0.6.14 (latest)
PyYAML: 3.09 (latest)
Python: 2.7.1 (latest)
wsgiref: 0.1.2 (latest)
All packages up-to-date
# pip upgrade PyYAML==3.08
Downloading/unpacking PyYAML==3.08
....
Successfully installed PyYAML
Cleaning up...
# pip upgrade --all -v
Packages installed at latest version:
pip: 0.8.2 (latest)
distribute: 0.6.14 (latest)
Python: 2.7.1 (latest)
wsgiref: 0.1.2 (latest)
Package updates available:
PyYAML: 3.08 (installed) 3.09 (latest)
Downloading/unpacking PyYAML
...
Successfully installed PyYAML
Cleaning up...
Removing temporary dir /root/upgrade_env/build...
Original Comment By: Kelsey Hightower
Last set of questions (I hope):
- Should pip upgrade parse the requirements file and filter out
editables? What about URL requirements?
For each non-editable item in the requirements file I would like to check the
indexes for a later version. In order to do this I would need to gather the
package info from each line in the requirements file.
Any tips are welcome (currently looking at pip.req.parse_requirements)
- Should pip upgrade search the indexes for a later version to install?
(This is what I am doing now). If not how should the upgrade command determine
if there is an upgrade?
Right now I am only adding packages to the upgrade list when:
- The package is not installed
- An upgrade is available (later version from the indexes), and no explicit
version was requested - The requested version is different from the installed one (Version miss
match). - I am deferring the requirements file to the install command until I filter
out the non-editable requirements.
Original Comment By: Kelsey Hightower
Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.
Original Comment By: Kelsey Hightower
Carl after re-reading your post, it seems I am doing more than what is
required. I will upload my branch so you can take a look. I may have went
overboard by checking PyPi for a later version.
Original Comment By: Kelsey Hightower
Yeah, it sounds like you're doing more than should be needed. Pip install
--upgrade does everything you're discussing already (checking for newer
versions, etc); all "pip upgrade" should be is like a two-liner passing
everything over to pip install --upgrade.
The only real code to be written here is the code for "upgrade --all" to get
the full list of installed upgradeable packages in the environment.
Original Comment By: Carl Meyer
Yeah, I knew it. Well, I did learn alot. Even though not required for this
task, I do like the ability to show whats installed and available during the
upgrade process (see test run above).
I will re-factor and clean things up. Current changes in my fork on the
upgrade-command branch.
https://bitbucket.org/khightower/pip/changeset/2bdc202b446c
Original Comment By: Kelsey Hightower
I have stripped down the upgrade command per Carl's suggestions (I went to far
in the first place). Not sure I like the results, but it does mirror install
--upgrade in functionality.
It seems pip tries to download and re-install the package even when the
package is already installed and up-to-date. Even worse with upgrade
--all, pip re-installs everything including pip itself. Is this how pip
upgrade should work? If so then I am almost done :)
Running pip upgrade command
# pip upgrade Mako
Downloading/unpacking Mako
Running setup.py egg_info for package Mako
warning: no files found matching '*.jpg' under directory 'doc'
warning: no files found matching '*.sty' under directory 'doc'
warning: no files found matching 'autohandler' under directory 'doc'
warning: no files found matching '*.xml' under directory 'examples'
warning: no files found matching '*.mako' under directory 'examples'
warning: no files found matching '*.dat' under directory 'test'
warning: no files found matching 'ez_setup.py'
Downloading/unpacking MarkupSafe>=0.9.2 (from Mako)
Running setup.py egg_info for package MarkupSafe
Installing collected packages: Mako, MarkupSafe
Found existing installation: Mako 0.3.6
Uninstalling Mako:
Successfully uninstalled Mako
Running setup.py install for Mako
changing mode of build/scripts-2.7/mako-render from 644 to 755
warning: no files found matching '*.jpg' under directory 'doc'
warning: no files found matching '*.sty' under directory 'doc'
warning: no files found matching 'autohandler' under directory 'doc'
warning: no files found matching '*.xml' under directory 'examples'
warning: no files found matching '*.mako' under directory 'examples'
warning: no files found matching '*.dat' under directory 'test'
warning: no files found matching 'ez_setup.py'
changing mode of /root/upgrade_env/bin/mako-render to 755
Found existing installation: MarkupSafe 0.11
Uninstalling MarkupSafe:
Successfully uninstalled MarkupSafe
Running setup.py install for MarkupSafe
building 'markupsafe._speedups' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -fPIC -I/opt/OpenPython-2.7.1/include/python2.7 -c
markupsafe/_speedups.c -o build/temp.linux-x86_64-2.7/markupsafe/_speedups.o
gcc -pthread -shared
build/temp.linux-x86_64-2.7/markupsafe/_speedups.o -o
build/lib.linux-x86_64-2.7/markupsafe/_speedups.so
Successfully installed Mako MarkupSafe
Cleaning up...
Original Comment By: Kelsey Hightower
Kelsey - Yeah, there are some bugs with install --upgrade; in particular you
identified #13, that it re-downloads and re-installs even packages that
are already up to date. The solution there isn't to do something different
with the new upgrade command, it's to fix the bug in install --upgrade.
With upgrade --all, it seems reasonable to me that pip would upgrade itself
too, if there's an upgrade available. (Personally I will never use upgrade
--all, so I don't know what behavior people who will use it would want from it
here). Obviously again it'd be better-behaved if #13 were fixed.
Original Comment By: Carl Meyer
If anyone has time please review my upgrade-command branch. In the meanwhile
I'll work on unittests that try not to duplicate the existing ones for the
install command.
https://bitbucket.org/khightower/pip/src/fa7b2a6d2bf1/pip/commands/upgrade.py
Original Comment By: Kelsey Hightower
@vababiy: i have tried your upgrade-command, but it seems not to work correctly... So i made a own one:
@jedie i think you meant to direct your comment at @KHightower. @vbabiy migrated her comment to here but did not write the upgrade command.
+1
@kelseyhightower Any progress?
+1
+1!
+1
+1
Please stop commenting on the issue with just a "+1". We're aware that the feature is wanted, spamming our inbox doesn't help though.
Instead I'd be thrilled to see a comment "patch done!" ;)
+1
Any Updates? Is there any plan to add this, it's 3 years old now..
+1
I thought this was merged already. I can dust off my python skill and give it another go.
Will this feature be included in 1.5? Can't find any reference to it in the 1.5 documentation...
+1
What is the status of this issue?
It is blocked by #988
If it's really important for you, there are workarounds for upgrading all packages; I threw together a script to do so in parallel(https://github.com/ariccio/update-pip-packages), and there are many other implementations elsewhere on the internet.
There's two parts to this issue. upgrade-all may be blocked by gh-988, but I don't see how upgrade is blocked. pip upgrade can be a simple alias for pip install -U --no-deps. This would resolve one of the main problems of using install_requires in setup.py files. Can't this be done sometime soon?
pip upgrade can be a simple alias for pip install -U --no-deps
from the description:
pip upgrade would be like pip install --upgrade except it would be non-recursive by default (and offer a --recursive option). It's current recursive default behavior has caused grief for many (#304). As for how to do non-recursive upgrades now, see here
that is not pip install -U --no-deps
“non-recursive” in this context does not simply mean –no-deps. A non-recursive upgrade will upgrade dependencies, but only if needed to fulfill parent requirements.
@qwcode thanks for the clarification. Then that's not what I care about. Why would you call this "non-recursive", it's still recursive but just a bit smarter/different?
I was under the impression from the discussion in gh-571 that really non-recursive was the desired default. That would certainly make sense, and prevent having to always write code like scipy/scipy@8e7ee0c4b3c16.
in #571, "non-recursive" is not --no-deps it's the "smarter/different" recursive as you say.
notice the test_upgrade_with_needed_recursive_upgrades() test from that PR.
without getting stuck on terms, there's 3 things:
- the "smarter/different upgrade", i.e. the kind that occurs in OS packaging that can also imply upgrading dependencies (but only if they actually need upgrading).
- what pip does now, which is to upgrade all dependencies, regardless of need or conflict resolution.
--no-deps
some possible ways to distinguish #1 and #2
- "non-recursive" vs "recursive"
- "normal" vs "recursive"
- "only if needed recursive" vs "do it regardless recursive"
I'm open to using the best terms... just not sure what those are.
I think I'm liking this "only-if-needed recursive" phrase. maybe i should use that here in the docs:
I like it too. If you'd describe the three options all together as
a. non-recursive
b. only if needed recursive
c. recursive (or "do it regardless recursive")
that would be clear.
Then you want to pick good defaults. For both upgrade and install -U, either (a) or (b) could make sense. I strongly prefer (a), but (b) makes sense too given that that's what OS packaging does.
(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U.
To clarify why I strongly prefer (a): an unsuspecting user wanting (b) and getting (a) will have to read a message saying "non-recursive upgrade can't satisfy all dependencies, please use only-if-needed recursive", which is not that big a deal. If the default would be (b), that unsuspecting user may end up with an upgrade or even a broken install of a package he really didn't want to be touching. That can take hours or even days to recover from.
(c) doesn't make any sense as a default, so please reconsider your decision not to change install -U
the reason to leave install -U alone is just for compatibility reasons, and then to eventually deprecate it.
If the default would be (b), that unsuspecting user may end up with an upgrade
or even a broken install of a package he really didn't want to be touching
if a user wants necessary dependency upgrades to go unfulfilled, they should specifically have to ask for that using --no-deps. there's no way in my mind that could ever be the default. that behavior would do more damage than what you're considering (which is the outlier case). Over and over again, users would be left not having the dependency upgrades they needed.
Deprecating install -U sounds good.
I agree (b) is more common than (a). Even if it would be 100x more common, which is not the case I think, more damage is untrue. Reading a clear error message before install begins is so much better than for example a compile error halfway through, that imho (a) is still the better default.
Relying on --no-deps may be fine for very experienced users, but new users are only going to learn about it after things go wrong.
Anyway, looks like I won't be able to convince you. Then back to the manual
try:
import dependency
except ImportError:
install_requires.append('dependency')
else:
if dependency.version < 'x.y.z':
raise ValueError('Please upgrade dependency to x.y.z')
it is.
@qwcode thanks for the detailed explanation. I hope this will be moving forward in the near future - only if needed recursive would be a huge improvement over the current behavior.
Spending again many hours this week fielding issues on why we don't use install_requires, I'm adding a 👍 to moving away from the current behavior.
Are there any updates on this issue?
We have now 2015 and I still need to copy from pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U stackoverflow
Yeah, essentially all the progress is linked to this issue.
GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)
Yeah for sure but I don't get the reason why this "simple" feature has such a delay.
What is the reason for this?
Am 16.04.2015 um 05:28 schrieb Alexander Riccio notifications@github.com:
Yeah, essentially all the progress is linked to this issue.
GitHub does quite a good job cross-referencing things. All of those are clickable! (I'm sure you know this?)
—
Reply to this email directly or view it on GitHub.
as for pip upgrade-all, the consensus seems to be to wait for work related to #988 before releasing any shiny new commands that are still ultimately flawed and can be dangerous to a working environment. A simple version of upgrade-all that doesn't properly resolve requirements can easily break existing packages
+1
What are you doing along 4 years?
+1
+1
+1
@muhasturk Currently, waiting... #59 (comment)
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1
+1 ...sorry for the spam, but running pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install -U hurts!
+1
In case anyone is interested in working on this, I think the comments above are somewhat confusing -- AFAICT the actual status is that pip upgrade-all is currently blocked by the need for a proper dependency resolution system, but pip upgrade could be implemented at any time. If someone wants to work on that part then it would be extremely awesome :-).
(See also the thread starting here and @dstufft's reply here and comment here agreeing with the above assessment.)
here's another discussion from the pypa-dev list from a year ago (that agrees that pip upgrade FOO could be done now) https://groups.google.com/forum/#!searchin/pypa-dev/pip$20upgrade/pypa-dev/vVLmo1PevTg/oBkHCPBLb9YJ
Thanks @qwcode! I also just saw the new description at https://pip.pypa.io/en/latest/user_guide/#only-if-needed-recursive-upgrade, that's helpful.
+1
If I'm not mistaken:
If xxx is not installed:
pip upgrade xxxwould be equivalent topip install xxxpip upgrade xxx --recursivewould be the equivalent ofpip install xxx --upgrade
If xxx is installed:
pip upgrade xxx --recursivewould still be the equivalent ofpip install xxx --upgrade(by design)- but there is currently no equivalent for
pip upgrade xxx, this could bepip install xxx --upgrade-only-if-needed/--upgrade-lazy
It's not clear what the added value of a new command is over adding a new option to pip install ?
It's not clear what the added value of a new command is over adding a new option to
pip install?
The default pip install -U behavior is unacceptable to many projects. See my comment above (#59 (comment)). And here's a longer explanation: http://article.gmane.org/gmane.comp.python.distutils.devel/24218
If it is unacceptable, then I guess you plan to change your usage of pip install --upgrade to pip upgrade ?
Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?
What does a new command provide that a new option could not ?
It's all paving cow paths. It's not @rgommers personal usage he's worried about, it's his users. People are going to reach for the "obvious" answer if they don't know better and right now the obvious answer is problematic for some major pieces of the Python ecosystem.
Indeed. People don't read docs. Indeed we'll fix all the install docs we can find, but as soon as a user sees -U or --upgrade that's what he'll use. The chance that people will use --no-deps or --only-as-needed or whatever before they've been seriously bitten by it is remote.
Why could you not instead change your current pip install --upgrade usage to pip install --upgrade-only-needed ?
And to be clear: we avoid pip install --upgrade now and do not use install_requires because of this. That's how bad it is.
I think there's a minor misunderstanding here -- IIUC @xavfernandez agrees with adding the non-recursive upgrade functionality, their question is just why the UI to that functionality has to be as a new top-level command instead of a new option to pip install.
@xavfernandez: Note that there's currently a discussion happening in #3194 about what would be the clearest UI for this functionality.
(And pip install -U will be deprecated and then removed, so that answers the question about how users will know not to use it :-).)
I think there's a minor misunderstanding here
I'm pretty sure that there isn't. And yes, we're only debating UI. But that's important.
Yup only debating the UI and I agree it is important. But what I see coming is that once pip upgrade is out, there will be no reason to keep using pip install...
And the UI to install any package will become "pip upgrade foo"
there will be no reason to keep using pip install
which is a problem why? The issue (at least for @qwcode in comments above) with changing pip install --upgrade to the right behavior is breaking backwards compatibility. So pip upgrade is the way to avoid that break and get the right default.
@rgommers: to be fair though, we will get some non-zero number of confused questions about why did you tell me to run pip upgrade foo when I don't even have foo installed, it didn't work (argue back and forth until they actually try running the command instead of assuming that it won't work and pretending that they ran it)
On #3194 I made the comment that maybe the right way forward is to just remove --upgrade from pip install, make pip install always do an upgrade of the explicitly named items, and default to a minimal upgrade strategy for dependencies with a --recursive flag to get the current behavior on upgrade.
I think I agree that I feel like with #3194 the default UX will become pip upgrade foo instead of pip install [-U] foo. The only problem I have with that is I think it's sort of confusing to have the two commands and given that I think pip upgrade would be the "right" one for people to use, I think it's crummy to have the obvious name be the wrong name.
I have a migraine so I could also just not be thinking entirely correct, but I'm sort of feeling like figuring out how to handle the deprecation path to that suggestion might be the right way forward.
Well, I won't be arguing that backwards compat is important here. I was always in favor of changing the behavior of pip install --upgrade. It's the cleanest, and the cost seems small. But it looked to me like it was vetoed by @qwcode, so pip upgrade was the next best thing (and already OK'ed by the pip devs a year ago).
If we now want to go back to changing defaults/behavior for pip install, great.
Since we agree to deprecate --upgrade, we could keep that and add two options to pip install: --upgrade-only-needed and --upgrade-eager , the second one being an alias for the deprecated --upgrade
@xavfernandez Forcing decisions onto users is kind of crummy I think. Unless I understand the fairly subtle issues I'm not sure that I would really know if I wanted --upgrade-only-needed and --upgrade-eager.
@xavfernandez so you're proposing that in the final situation the command would then be pip install --upgrade-only-needed? Seems ugly.
It would be much better to have something that maps to the equivalent of the semantic version pinning available in e.g. npm, hex, or Cargo. That would certainly need adjustment in the Python context, since (a) PEP 440 versions do not and cannot map precisely to semantic versioning and (b) the Python community at large doesn't necessarily hew to semantic-like versioning in its releases even within PEP 440.
Nonetheless, something along those lines would be very helpful, as would the notion of pinning a specified version.
Given the constraints, in the short term one viable option might be doing something analogous to homebrew's upgrade and upgrade --all commands, where the latter simply goes ahead with upgrading everything (potentially with a warning the first time).
In the long term, however, it would be enormously useful to create shared conventions about what versions mean in the Python packaging community, and building pip support around that could be a big part of it.
The obvious migration path would be:
pip version X:
- add options
pip install --eager,pip install --no-eager, where--eagermeans that for requirements with no version restriction attached we attempt to install the latest version even if there is already some version installed, and--no-eagermeans that we're satisfied if any version is installed - also add option
pip install --eager-recursivewith the current-Usemantics --no-eagerremains the default- if none of these options are specified, and
pip installis passed a requirement with no attached version number, and this requirement is already satisfied, then output a warning saying that we didn't upgrade your package, but in the future we will, so you should use--no-eagerif you want to keep the current behavior - add a deprecation warning to
pip install -Ureferring people topip install --eager-recursive
pip version Y:
- switch the default for
pip installto--eager
pip version Z:
- remove
pip install -U
I made Y and Z different versions because I am eager for --eager so maybe we could make that change on a more aggressive schedule, while keeping pip install -U around for a bit since a lot of documentation already refers to it, and it doesn't cause any harm. But obviously Y = Z is an option :-)
@chriskrycho: That all sounds like great stuff to have, but this discussion is about how to handle the situation where the user is explicitly trying to request an upgrade to the latest version of a package, which is a situation that commonly happens already today and we don't have a good answer to. I don't know if there's a bug open requesting pinning support yet, but if not then maybe you should start one?