pypa/pip

pip 20.3 release (Q4 2020)

pradyunsg opened this issue · 30 comments

pradyunsg commented

Another quarter! Another release!

We're definitely not skipping this release, because we have a new resolver to roll out -- and lots of really nice other enhancements as well.

pradyunsg commented

As always, need a release manager (RM) for this from @pypa/pip-committers.

I've self-assigned myself as the RM for this release (#7531 (comment)) -- as usual, if anyone is opposed to this, let me know. :)

  • 👍4
brainwane commented

Per our discussion in our meeting today, @pradyunsg believes we should be able to make this release in the second half of October, assuming we don't find a showstopper in a bug report before then.

brainwane commented

I just spoke with @pradyunsg about the timing of the 20.3 release. Conclusion: @pradyunsg and I are leaning towards modifying the plans for 20.3 to:

  • a pre-release this week
  • a stable release in mid-November, around Nov. 11-12

Reasons: In terms of features and showstoppers, we are set to be ready to release this week. However, a few things have come up, just now, at the end of the 20.2 beta period:

  • #8664 (comment) We need to nail down our policy and implementation re: Python 2 support.
  • #9030 and pypi/warehouse#4321 (comment) We've just run into some CI blips/glitches that have gotten in our way and are not yet completely resolved.
  • As Election Day in the United States (Tuesday, 3 November) approaches, we're seeing an increasing risk of civil unrest in the US (which may occur approaching, on, and immediately following 3 November), and we're considering the risk that a disruptive change to pip, released only a week or less before the election, would make it harder for civic tech developers/administrators and data scientists to use Python (in tools to assess/counter voting suppression and violence).

We planned an October release, in accordance with the usual release cadence, before the full scope of the pandemic and the US election situation emerged. But, as the release policy states,

The release manager may, at their discretion, choose whether or not there will be a pre-release period for a release, and if there is may extend that period into the next month if needed.

@pypa/pip-committers heads-up in case you are inclined to agree or disagree.

  • 👍11
  • ❤️2
brainwane commented

Looks like we should expect the prerelease in the next couple days (cc @pradyunsg).

brainwane commented

We're being blocked (in making the 20.3.beta1 release); it's held up by a Python 2.7-specific CI failure that Pradyun is unable to reproduce locally. So 20.3.beta1 may not be out till Saturday.

https://dev.azure.com/pypa/pip/_build/results?buildId=29831&view=logs&j=f801bdd4-2ff4-5da8-3bd6-deea013b4470&t=89b17dec-01ee-56ca-032e-26fbc447775d&l=70

pradyunsg commented

Alrighty, that's resolved now (see #9019 for details, and a very-annoyed-at-CI-stuff Pradyun). I'll be making the release (20.3.beta1) in the next few hours!

pradyunsg commented

pip 20.3 beta1 is up on PyPI. get-pip.py has not been updated.

Happy Halloween folks! :)

  • 🎉4
  • 🚀3
brainwane commented

Here's a rough plan for publicizing this release (separate from the script or checklist for actually making the release and uploading it to PyPI and updating get-pip.py):

  1. pypi-announce
  2. python-announce
  3. wheel-builders
  4. python-list
  5. psf-community
  6. psf-volunteers
  • 👍6
brainwane commented

@pypa/pip-committers @pypa/pip-helpers @di Heads-up: we decided in today's meeting that we will probably be releasing 20.3 tomorrow or Friday. We'll of course know more tomorrow.

brainwane commented

Pradyun has been chasing down several issues that have proven nastier or more critical than they originally seemed, notably #9011. (See the milestone.) We may be able to cut the new release on Sunday or Monday. I would really hate for it to be delayed much longer than that, such as going into December, but if we release something with known regressions that will create hard-to-debug trouble for enough users, then that'll cause high support costs for us and lowered credibility for future releases.

  • 👍10
pelson commented

@brainwane - just wanted to say that although sometimes it might feel like you are speaking to the void with these updates, in reality they are extremely helpful for downstream planning and long may they continue! Keep up the great work (and to all the pip devs!) 👍

  • 👍13
  • ❤️8
duckinator commented

I 100% agree with @pelson. It's so nice that I can just go to the issue tracker, find an issue about the pending release, and just... learn exactly what the state of it is. I can't think of any other project that does that so reliably. 🙂

  • 👍4
webknjaz commented

Sounds like it's fixed in the resolver upstream: #9011 (comment) 🎉

Have you considered having another beta/rc release before going final?

brainwane commented

The team was able to finally resolve #9011. In the past few days we also dealt with a new issue that's cropped up, around Mac OS Big Sur support,* and reviewed a few additional new or newly updated issues filed in the resolver.

IT LOOKS REALLLLLY LIKELY that we'll release pip 20.3 tomorrow, Monday, 30 November. @pradyunsg plans to publish it to PyPI around noon UTC/17:30 Mumbai/noon London/7am EST/4am PST on Monday. I'm writing release announcements today.

@pelson and @duckinator thank you. I really appreciate that.

@webknjaz We thought about it, but we do not have time to run a proper feedback cycle and listen and get feedback that might push the release date out further. So we may publish a beta in the next few hours, just in case there are people watching this & related issues who grab it and might find giant showstopper bugs, but we won't be publicizing another beta or release candidate before publishing 20.3 tomorrow.

* @pradyunsg and I discussed what to do about Big Sur support in pip 20.3. We had three options:

  1. Release pip 20.3 while completely leaving the status quo situation in place, which breaks things for some people who have upgraded to Big Sur and are attempting to use pip with default configuration (for instance, pip will attempt to compile numpy instead of using a wheel, and this will break because most people don't have the necessary dependencies).

  2. Merge some improvements to support Big Sur for most people #9170 (comment) , and break a few tests for now, and unfortunately leave a certain chunk of behavior broken for users who set configuration to claim they are on a different/specific architecture on MacOS (behavior fixed in pypa/packaging#361). Then release pip 20.3.

  3. Wait for Pradyun's packaging.tags pull request pypa/packaging#361 to be reviewed and merged, then get a new release of packaging, and vendor that into pip, and only then release 20.3.

To compromise between support and speed, we decided to go with option 2. See #9170 (comment) .

  • 👍2
  • 🎉8
  • ❤️4
  • 🚀6
joelberkeley-secondmind commented

I've been using pip check to check my pip installs have worked out. Does the new resolver make that unnecessary now?

brainwane commented

@joelberkeley-secondmind Unfortunately, no, not yet, because of an issue we're talking about in #9094.

joelberkeley-secondmind commented

ok, thanks. But it isn't necessary if the whole workflow is pip install x y z; pip check?

uranusjr commented

If you start with an empty environment, yes.

  • 👍1
pradyunsg commented

pip 20.3 has been released!

  • Uploaded to PyPI.
  • Pushed updates/tags to GitHub.
  • get-pip.py has been updated, and the change will propagate over the next few minutes.
  • 👍9
  • 🎉6
  • ❤️5
  • 🚀4
groodt commented

@pradyunsg @uranusjr and the many others involved. Thank you! What a huge achievement!

  • ❤️7
brainwane commented

We're planning a 20.3.1 release for some followup bugfixes.

I went most of the way through the release publicity checklist I made in #8936 (comment) .

I screwed up -- I forgot to mention in our announcements and other materials that the old resolver is still the default when using pip with Python 2.x.

I am going to:

  • update the blog posts
  • update the Discourse post
  • send an update to distutils-sig
  • send an update to python-list
  • send an update to python-announce - deciding whether or not to do this
  • send an update to wheel-builders
  • update #6536
  • send an update to python-dev
  • see what needs updating in the user guide docs at pip.pypa.io and submit a pull request - done, #9269
pradyunsg commented

There's a 20.3.2 release that I'd likely make this week (and likely a 20.3.3 release).

@pypa/pip-committers Please don't merge non-bugfix stuff until 20.3.2 is out, since I'd like to be lazy and cut the release directly off of the default branch instead of the tag. :)

pradyunsg commented

There's a 20.3.2 release that I'd likely make this week (and likely a 20.3.3 release).

Done!

  • Uploaded to PyPI.
  • Pushed updates/tags to GitHub.
  • get-pip.py has been updated, and the change will propagate over the next few minutes.

@pypa/pip-committers consider the master branch to be open for pip 21.0 now. I'll cut the remaining bugfixes via cherry-picked commits.

  • 👍3
  • 👎1
  • 🎉3
  • ❤️2
  • 🚀2
pradyunsg commented

For anyone following along, pip 20.3.2 was yanked, and pip 20.3.3 will be released today, fixing the underlying issue that led to us yanking pip 20.3.2.

See #9293 for relevant details.

  • 👍1
  • 🎉1
  • ❤️1
  • 🚀1
  • 👀1
hugovk commented

As an aside, I'm a little surprised that pip install -U pip installed the yanked version (albeit with a warning):

$ pip install -U pip
Requirement already satisfied: pip in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (20.3.1)
Collecting pip
  Downloading pip-20.3.2-py2.py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 2.2 MB/s
WARNING: The candidate selected for download or install is a yanked version: 'pip' candidate (version 20.3.2 at https://files.pythonhosted.org/packages/3d/0c/01014c0442830eb38d6baef0932fdcb389279ce74295350ecb9fe09e048a/pip-20.3.2-py2.py3-none-any.whl#sha256=8d779b6a85770bc5f624b5c8d4d922ea2e3cd9ce6ee92aa260f12a9f072477bc (from https://pypi.org/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*))
Reason for being yanked: <none given>
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.1
    Uninstalling pip-20.3.1:
      Successfully uninstalled pip-20.3.1
Successfully installed pip-20.3.2

The docs say:

A yanked release is a release that is always ignored by an installer, unless it is the only release that matches a version specifier (using either == or ===). See PEP 592 for more information.

https://pypi.org/help/#yanked

Is this a bug?

  • 👍1
  • 👀1
pradyunsg commented

Yup. That's a bug in the new resolver's handling in pip 20.3.1, that was fixed in 20.3.2. Sadly, pip 20.3.2 is borked in a different way. :)

  • 👍3
pradyunsg commented

For anyone following eagerly -- #9302 is the PR for the 20.3.3 release. Once CI says "OK to go", I'll upload it. :)

  • 👍1
  • 🚀1
pradyunsg commented

20.3.3 is out!

  • Uploaded to PyPI.
  • Pushed updates/tags to GitHub.
  • get-pip.py has been updated, and the change will propagate over the next few minutes.
  • 👍5
  • ❤️2
  • 🚀3
pradyunsg commented

21.0 is imminent, so I'll cut the 20.3.4 sometime in the next few days.

Note that (unless something really weird happens), 20.3.4 will be the last release of 20.3.* series, and the last Python 2-compatible release of pip.

pradyunsg commented

20.3.4 is out!

  • Uploaded to PyPI.
  • Pushed updates/tags to GitHub.
  • get-pip.py has been updated, and the change will propagate over the next few minutes.

Calling that the end of 20.3's lifecycle. Expect a 21.0 today/tomorrow. :)

  • ❤️2
  • 🚀4