pip-review?
gavinjackson opened this issue · 33 comments
Hello,
Thanks for all the great work on pip-tools! I like the way that the project is heading (pip-compile and pip-sync look pretty cool).
I just noticed that the latest release has removed pip-review. I was wondering what the new equivalent is (as I found this tool very useful)?
eg. every night I build a new pyvenv, install required pip libraries and run pip-review to send an email out to the devs letting them know if any libraries require upgrading.
Could pip list -o work for you?
Else, have a look at piprot?
https://github.com/sesh/piprot
Thanks, that's great!
This means we no longer have a replacement for pip-review --interactive. The rationale for removing pip-review is that pip-compile + pip-sync are enough, however that means that we need to use pip-compile, something I'm not yet ready to do.
@nvie Is there any hope to get pip-review back, eg. "patches welcome"? Or is it decidedly incompatible with the vision?
I don't think it's useful. If there's a specific flow that depends on the tool, let's see if we can figure out an alternative.
Either you could run pip-compile (and check the git diff) or run pip-compile --dry-run (and diff it to your current requirements.txt yourself), or run pip list --outdated (to check your venv for outdated packages).
I do agree that the guided interactive prompting would be a nice to have, but I don't think we need to bring back pip-review for it. Perhaps we could think about a pip-compile --interactive instead, but I'd need to think about more details here.
The workflow really is "interactively update a pinned requirements.txt file". If pip-compile --interactive could do it, I'd be fine with it.
I think I'll end up using requirements.in at some point, so if nobody else wants interactive review, it's possibly not worth it.
I like it and use it regularly.
Sorry. I liked it, and used it regularly.
I use pip-review to keep my virtual environment packages up to date and not to manage packages for a particular project. I don't really have a need to pin my packages and don't find pip-compile helpful since I'm not tied in to a specific package version / upgrade path.
Could you bring back pip-review for use cases, like mine, where we are purely looking to upgrade all packages without pinning them? Thanks!
+1, was using pip-review to upgrade packages easily.
I'd be more than happy having pip-review back :(
Unless I'm missing something, I think you can all use pip list --outdated instead, which has gotten much better lately, and is also maintained. Today, it basically provides the same features, and has replaced the need for pip-review entirely.
pip list --outdated doesn't upgrade packages, though, does it? We still need to upgrade them one at a time. With pip-review --interactive or auto, I can instruct the process to upgrade all packages at once.
Before:
$ pip-review --interactive
Yes/No/All/Quit * 12
After:
$ pip list --outdated
[list of 12 pip outdated packages]
$ pip install -U package1
$ pip install -U package2
...
$ pip install -U package12
I can see the convenience in that, but it's not enough to keep around a separate project for and maintain it even if pip already offers most of this. Since the only added value is the convenience of prompting for upgrades, I suggest you script around pip list for that. A rudimentary simple way of doing this in a shell script that would basically bring back pip-review is this:
pip list --outdated | while read pkg; do
echo $pkg
echo 'Update now? [yn]'
read answer
if [ "$answer" = "y" ]; then
pip install -U $(echo "$pkg" | cut -d' ' -f 1)
fi
doneThat's what I was thinking as a substitute - point.
@nvie i guess this is mostly a problem of handling deprecation which impacts peoples' workflows... this thread is pretty hidden and so most developers who update pip-tools and used pip-review before will suddenly think "wat?"...
slowly deprecating it with a warning would've been a bit nicer, but seems it's too late for that now... maybe you could include a few words about pip-review being dropped in favor of pip list --outdated or even the script? That would be quite easy for you and save many people some time i guess...
Note that the script above didn't work for me.
This did:
#!/bin/bash
for pkg in $( pip list --outdated | cut -d' ' -f 1 )
do
echo $pkg
echo "update now? [yn]:"
read answer
if [ "$answer" == "y" ]; then
pip install -U $pkg
fi
doneIt'd be great to have a note in the README about pip-review being removed and the alternative.
Script with current/latest version:
#!/bin/bash
while read line; do
test -z "${line}" && continue;
echo ${line};
pkg=$(echo $line|cut -f 1 -d' ');
echo -n "Upgrade now? [y/n]: ";
read answer </dev/tty;
test "${answer}" == "y" && pip install -U ${pkg};
done< <(pip list --outdated)The removal breaks Upgrading all packages with pip use-case.
Fundamentally changing the tools provided by a project isn't so much a deprecation as a total project re-purpose.
It might have been better to leave this repo (and pypi entry) as-is (and marked as "no longer developed") and create a new repo for pip-sync.
It would behoove someone to fork this repo and rollback the depreciation.
I am a little confused and maybe the is not the place to voice this concern. My main use of pip-tools was to keep my cutting edge virtualenv up to date with the latest packages. The removal of pip-review has broken this workflow.
An additional note, how does the new system in anyway differentiate itself from the python recommended method of pip install -r requirements.txt -U. If I now have to keep another requirements.txt for my bleeding edge virtualenv, why would I use an additional third party library?
I am really unsure as to what benefit the refactoring of this package brings over what pip already offers?
@bnice5000: for the best rationale behind the tool over pip install -U, see this blog post I wrote up a while ago: http://nvie.com/posts/better-package-management/
In case, you want to update all at once without having to press y all the time (mostly if your project has lots of dependencies, use this:
#!/bin/bash
for pkg in $( pip list --outdated | cut -d' ' -f 1 )
do
echo $pkg
pip install -U $pkg
doneI went ahead and created a standalone package for pip-review: https://pypi.python.org/pypi/pip-review
Please don't be shy and submit as many pull requests as you can manage to my fork: https://github.com/jgonggrijp/pip-review
The initial release is called version 0.3.7, but the only difference with 0.3.6 is that I removed pip-dump (which is obviously obsolete now that we have pip-compile and pip-sync) and changed the name and maintainer of the package.
That's great, thanks!
Sorry to bring this issue back up, but I did find a use case for which pip list --outdated is not great. Form what I understand, the main point of separating the requirements into the .in and the .txt file is so that we have a more maintainable file. I keep all my pinned dependencies in requirements.in.
When I want to update the dependencies of my project, I only want to see the newer versions of the packages that are listed in requirements.in, not of every one installed in the environment. The assumption is that I can go over upgrading the main ones, and let pip figure out the necessary upgrades to all the packages.
Do this make sense? Is there a way of using the tools that are in the this project to address this point?
You are probably looking for pip-compile --upgrade.
Not quite, because I'd like to see the packages that needs to be updated, and manually update them. My understanding of pip-compile --upgrade is that it directly updates everything to the latest version, right?
My understanding is that pip-compile --upgrade does not actually upgrade anything. It just pins the latest available versions of your main requirements as well as the latest compatible versions of their dependencies in your requirements.txt, instead of the versions that are currently installed. So you end up with a requirements.txt that, if installed using pip-sync or pip install -r, will cause your main requirements and their dependencies to be updated.
FWIW, I believe that you are not supposed to manually pin your requirements in the requirements.in. You should just list them (without versions) and leave the pinning to pip-compile. The requirements.txt should be the single source of truth about what versions of what packages you rely on.
As far as I know, there is no tool that lists only the outdated primary requirements and lets you update them interactively. I think the following workflow comes nearest: run pip list to see the installed versions and pip-compile -Un to see the latest versions. The latter command states explicitly whether a package is a primary requirement or an indirect dependency, so you can use that to mentally filter the information. Selectively upgrade primary requirements using pip install -U. Finish by running pip-compile (without -U and without -n) to pin the new state of your virtual environment. Optionally, run pip-sync after this to remove packages that you don't need anymore.
Personally, I tend to use a lazier workflow: simply upgrade everything using pip-review --auto (from the separate pip-review package). Most of the time, everything still works fine after the update and you can directly pin the new state using pip-compile. Some obsolete packages might remain in your environment but pip-compile won't list them, so you don't need to be afraid that you'll install "garbage" when later using pip-sync. If something breaks, fix the problems before pinning, or when there are too many problems, revert to the pre-update state using pip-sync and then only install the most urgent updates.
@jgonggrijp Thanks for sharing your workflow. What you are saying makes a lot of sense. I think I initially had only the package names in my requirements.in. At some point, I added a single new package to my requirements.in, ran pip-compile, and saw a whole of changes (that I didn't want to deal with then). This is when I started also pinning some of the packages in requrements.in.
But I think I found a solution to the above problem. I also use piprot, and run piprot requirements.in to show me the outdated files. I can then selectively decide to update packages as I see fit, by making manual changes. For instance, I'd like to immediately implement a minor update to psycopg2, while updating from django 1.9 to 1.10 is a bigger task.
I wonder if this breaks some patterns that pip-tools is supposed to address?
Yes, I think the workflow with piprot that you describe is pretty much against the philosophy behind pip-tools and pip-review. Partly because requirements.in is not at all supposed to look like a pinned requirements.txt, and partly because the point of pip-tools is that you start with a working environment and then pin it in order to ensure that the exact same environment can be reproduced elsewhere in full working order (see pin your packages and better package management). You should first check that a new version works and then pin it, not the other way around.
Frankly, I don't see much place for piprot in a workflow based on pip-tools. I can only think of using it as a diagnostic; if piprot reports that your requirements.txt is 300+ days outdated, that might be a reason to consider putting some effort into updating. However, I think requires.io is a more flexible, powerful and convenient tool for such diagnostics.
The reason that you describe, for starting to pin versions in the requirements.in and using piprot, might be outdated. ;-) In the past, pip-compile always implied --update, so that might explain why you were seeing unwanted changes. Nowadays, pip-compile only pins installed versions by default. So you might be able to do away with piprot entirely.
To whom it concerns: pip-review reached version 1.0 (PyPI). It has become a thin wrapper around pip list --outdated that still enables you to update your packages automatically or interactively.
@nvie pip-review has been ostensibly under BSD license since before I adopted it. I figured it might be time to make it official (see jgonggrijp#56). Do you agree if I add the 3-clause BSD license text to the pip-review repository with the following header?
Copyright 2012-2015 Vincent Driessen
Copyright 2015-2017 Julian Gonggrijp
pip-review is still a nice tool to upgrade python package system wide but it will stop if a package fail to install.
I recommand pipdate instead : https://github.com/nschloe/pipdate