pypa/pip

Requested (package) has different version in metadata

potiuk opened this issue · 41 comments

potiuk commented

Describe the bug

In Airflow, we are experiencing problems when we are trying to use the PIP released 2 days ago which has the new resolver on by default.

ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow==1.10.13) has different version in metadata: '1.1.0'

How to reproduce

  • Setup an empty virtualenv for Python 3.6
  • Upgrade to latest pip: pip install --upgrade pip
  • pip --version should return 20.3
  • Run this command:
pip install "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt
  • Observe the output. It will keep on finding good dependencies until
Collecting oauthlib[signedtoken]>=1.0.0
  Using cached oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Using cached oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Using cached oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Using cached oauthlib-2.0.6.tar.gz (127 kB)
  Using cached oauthlib-2.0.5.tar.gz (129 kB)
  Using cached oauthlib-2.0.4.tar.gz (127 kB)
  Using cached oauthlib-2.0.3.tar.gz (127 kB)
  Using cached oauthlib-2.0.2.tar.gz (125 kB)
  Using cached oauthlib-2.0.1.tar.gz (122 kB)
  Using cached oauthlib-2.0.0.tar.gz (122 kB)
  Using cached oauthlib-1.1.2.tar.gz (111 kB)
  Using cached oauthlib-1.1.1.tar.gz (108 kB)
  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

Apparently, metadata in published oauthlib 1.1.0-1 is wrong snd points to 1.1.0.

The same command with the legacy resolver works fine:

pip install --use-deprecated legacy-resolver "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

Expected behavior

I expect the resolver does not get broken by broken metadata.

I've opened similar issue oauthlib as I am not sure who can fix it: oauthlib/oauthlib#744

brainwane commented

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

I expect the resolver does not get broken by broken metadata.

I understand that this was the previous behavior, but pip's maintainers have decided to make pip stricter and more consistent in dealing with incompatible, inconsistent, and broken combinations of metadata. We are doing this to pave the way for a lot of deeply-needed bugfixes and features, such as warning the user when uninstalling a package that other packages depend on, adding an "upgrade-all" command to pip, properly respecting constraints, and more.

I'll defer to others on diagnosing precisely what needs to be fixed in order to resolve this. Thanks again.

  • 👍1
notatallshaw commented

I feel like this is still a bug with the new pip resolver. Reproducing it myself I get the same error (the previous line included):

INFO: pip is looking at multiple versions of oauthlib[signedtoken] to determine which version is compatible with other requirements. This could take a while.
Collecting oauthlib[signedtoken]>=1.0.0
  Downloading oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Downloading oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Downloading oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Downloading oauthlib-2.0.6.tar.gz (127 kB)
  Downloading oauthlib-2.0.5.tar.gz (129 kB)
  Downloading oauthlib-2.0.4.tar.gz (127 kB)
  Downloading oauthlib-2.0.3.tar.gz (127 kB)
  Downloading oauthlib-2.0.2.tar.gz (125 kB)
  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

When searching for a version of a package that is compatible with other requirements Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

If no version with valid metadata can be found that's what Pip should be reporting, not that a very old version of the package had broken metadata.

  • 👍1
uranusjr commented

Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

We complete agree.

pip/src/pip/_internal/resolution/resolvelib/candidates.py

Lines 204 to 214 in 30eeb9c

def _check_metadata_consistency(self, dist):
# type: (Distribution) -> None
"""Check for consistency of project name and version of dist."""
# TODO: (Longer term) Rather than abort, reject this candidate
# and backtrack. This would need resolvelib support.
name = canonicalize_name(dist.project_name)
if self._name is not None and self._name != name:
raise MetadataInconsistent(self._ireq, "name", dist.project_name)
version = dist.parsed_version
if self._version is not None and self._version != version:
raise MetadataInconsistent(self._ireq, "version", dist.version)

There are, however, still may issues in pip that needed to be worked on, and this particular one is relatively low-impact and have alternative solutions. I understand it may feel urgent when it directly affects you, but there are really only a very small number of packages that have this issue (I believe this is the only one we know served on PyPI so far), and the package maintainer can resolve the issue by yanking the problematic release. The issue is therefore currently considered less urgent.

  • 👍2
  • 👎1
potiuk commented

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

Very good question indeed. The problem in our case that for a long time we had conflicting requirements in Airflow (we have > 420 requirements in total, because Apache Airflow is an orchestrator and we have more than 70 different providers (GCP/AWS/Postgres etc.) - each with its own set of requirements. We are releasing Airflow 2.0 (Release candidate planned for next week) and we've worked hard to get the 2.0 out, and only last week I started to look at our deps to make them conflict-free. I managed to do it (and now we have automated system that keeps the dependencies automatically updated but also pip-checked - those depencies are going to be upgraded every time after they pass all tests and pip-check in our master build. We are using automatically generated set of "good" constraints : https://github.com/apache/airflow/tree/constraints-master - if you look at the history of those commits, you will see that they get automatically updated whenever pip install .[all] --upgrade --upgrade-strategy eager will produce a set fo constratins that:

  • all tests pass with them
  • the constraints pass the 'pip check` (this works as of last week - previously we had conflicts in those constraints)

We could not test the new resolver before we fixed the constraints, because we had those conflicting constraints. With those, the new resolver went into an endless loop :(. Which is of course totally understandable. Since I just fixed the constraints, that was really the first time we could run the new resolver. And it is quite a coincidence of schedule with the PIP 20.3 release and Airflow 2.0 release as we've already in the past also had some limitation w/regards of PIP for a while and we had to pin it iand update our documentation.

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

Thanks!. This is what we've done - we updated the documentation and told our users to use this for now (or downgrade to the previous version of PIP). In our official docker images, we pinned PIP to 20.2.4 for now. I would love to use the new resolver thought and wondered if there is a way to disable the strict check :). But i understand from the discussion that it is a design decision for now. I understand that there are more pressing issues, so we can go with that constraint for a while - not ideal and caused us a spike of issues and questions from our users already, so we would prefer if this is fixed of course, also maybe the oauthlib team will fix it (for example by removing the offending version?).

potiuk commented

Another input:

I looked how we can workaround it and I thought we can do it by limiting the oauthlib directly, but it seems we can't:

We already have this in our setup.py oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2 . It's a bit strange that in this resolver downloads "all versions" of the lib even if we have this limitation (>=1.1.2):

  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'
potiuk commented

Another question. Would removing or yanking the 1.1.0-1 of oauthlib version help ? I proposed this in oauthlib/oauthlib#744

uranusjr commented

Yes, I’d say yanking is the best solution for now.

  • 👍1
ashb commented

Does this error (where a single bad dist causes the resolver to stop) not leave the python ecosystem open to a "left-pad attack" -- a disgruntled package author of something used by lots of transitive deps uploads a purposefully broken dist, 💥 half of pypi becomes uninstallable.

  • 👍4
  • 🚀2
  • 👀4
uranusjr commented

With any luck, we’ll be in time to release a solution before it happens. But even if not, I believe the PyPI terms of use give the admins power to remove things uploaded in bad faith. This is slightly different from the left-pad problem since removal is within a user’s rights and the admin cannot restore it without consent. (Disclaimer: IANAL and do not involve in maintaining PyPI.)

  • 👍1
notatallshaw commented

but there are really only a very small number of packages that have this issue (I believe this is the only one we know served on PyPI so far),

This is reasonable why this particular issue is lower priority, and I totally understand the difficult situation pip is in here.

and the package maintainer can resolve the issue by yanking the problematic release. The issue is therefore currently considered less urgent.

This is a little problematic in principle, if there is a failure in the pip resolver that a 3rd party package maintainers must resolve for my package dependencies to be resolved via pip this leads to a concern in the stability of using pip to handle such dependency resolution.

IMO, there should be a way to specify package versions from the pip dependency resolver ever having to consider, just as one can specify package versions to not be installed. Particularly as the resolver needs to download and extract each package version to assess the dependency requirements, which can explode the number of downloads and extraction required when dealing with large dependency trees.

In fact it's odd to me that resolver doesn't use the install specification to proactively cull possible package versions, e.g. when oauthlib!=2.0.3 is specified in the initial package requirements the resolver still downloads and extracts this version of the package. Is this also a #TODO? Or is there some existing solution we are all missing here?

I can understand though that the number of packages with large dependency trees is small and therefore lower down on pips priority list. Looking forward to see the resolver improve though and thanks for all your hard effort in getting it to provide at least some guarantees 😺 .

pfmoore commented

IMO, there should be a way to specify package versions from the pip dependency resolver ever having to consider

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

Is this also a #TODO?

Pip considers requirements and dependencies one by one, removing specific cases as we discover reasons they are invalid. The order in which we check things is theoretically irrelevant, but in practice can have a significant effect on how much work we do. We're working on improving that (#9185) but it's balancing a number of heuristics, so it's never going to be perfect for everyone.

What we don't have is a way to say "pretend version X of package Y simply doesn't exist, anywhere, ever". Users can simulate this by running a local PyPI proxy that strips that version out, but that's a very heavy-handed solution. It would be possible to add such a feature to pip (in effect, a config file that tells pip's "finder" to discard specific package/version combinations), but we don't have such a thing at the moment. I don't honestly know how many people need something like this (we're still trying to triage and categorise the various issue reports in this area) so at the moment I have no idea how much priority such a feature would get, but it's a possibility.

notatallshaw commented

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

I think I'm misunderstand what you mean by this or it doesn't work, I tried adding the following to the constraints file (which already has oauthlib==3.1.0):

oauthlib!=1.1.0
oauthlib!=1.1.0-1

And I still get the error:

  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

Pip considers requirements and dependencies one by one, removing specific cases as we discover reasons they are invalid. The order in which we check things is theoretically irrelevant, but in practice can have a significant effect on how much work we do. We're working on improving that (#9185) but it's balancing a number of heuristics, so it's never going to be perfect for everyone.

Thanks! Yes and understood. I followed the improvements of the conda resolver that faced similar issues (and it was still difficult in a more controlled environment no less), I totally appreciate what a difficult problem it is and pips work here is great.

potiuk commented

There is. You can add a constraints file with a line mypackage != 1.0.0 and pip will not consider that specific version of the package.

@pfmoore I can confirm it does not work this way.

We already had 'oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2, in our setup.py. I even explicitly tried earlier today (hoping that we can workaround it) with both != 1.1.0 and != 1.1.0-1 and the installation broke the same way trying to download 1.1.0-1 in both cases. Seems that PIP downloads in this particular case ALL versions no matter what constraints it had.

Just one more comment -> the constraints we have are for sure GOOD. They were prepared using 'pip freeze` from a working installation of Airlfow and they pass ''pip check' of the pip 20.2.4 , so I assume they are fully consistent, and whatever PIP 20.2.4 installed, pip 20.3 should as well. It's also concerning why those packages are downloaded AT ALL. From how I understand the new resolver works, the --constraints flag works differently than in pip 20.2 - it basically limits the version "visible" to the resolver. So the resolver should not even know anything about oauhtlib different than 3.0.1, because the constraint file of ours:

https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

contains:

oauthlib==3.1.0

This is at least what I understood from your comment here: #8210 (comment)

This makes me a little worried about soundness of the design and stability of the new pip in general.

Why those different versions are downloaded in the first place?

  • 👍2
potiuk commented

The maintainers of oauthlib were so kind that they yanked the offending release. @pfmoore, unfortunately it did not help.

I see the release yanked, but still have the same error :(

terrdavis commented

fyi, i encountered the same issue using a local simple index (so the package may be very out of date).

ERROR: Requested pypyodbc from file://[...]/packages/simple/pypyodbc/pypyodbc-1.3.5.2.zip has different version in metadata: '1.3.4'

For now, I'll just pin pip to 20.2.

uranusjr commented

I think the != clause needs to be added as a top-level requirement, or in a constraint file instead, otherwise pip may try to resolve other packages first (and fail), because the != specifier hasn’t been seen yet when it hits the invalid version.

notatallshaw commented

I think the != clause needs to be added as a top-level requirement, or in a constraint file instead, otherwise pip may try to resolve other packages first (and fail), because the != specifier hasn’t been seen yet when it hits the invalid version.

As explained in my comment above I tested adding the != in the constraint file, it didn't stop the package being downloaded and extracted by the resolver and I got the same error.

notatallshaw commented

The maintainers of oauthlib were so kind that they yanked the offending release. @pfmoore, unfortunately it did not help.

I see the release yanked, but still have the same error :(

Confirmed, I see 1.1.0 is yanked: https://pypi.org/project/oauthlib/1.1.0/

I cleared my pip cache and re-ran the test, same error:

  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
     |████████████████████████████████| 106 kB 6.4 MB/s
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

So it seems the new pip resolver is also not respecting the Yanked status of a package 😞 .

I did notice that 1.1.0-1 was downloaded after 1.0.3, 1.0.2, 1.0.1, and 1.0.0 which is a different behavior than before where 1.1.0-1 was downloaded before those versions, but ultimately the error is the same.

pfmoore commented

Version 1.1.0-1 is different than 1.1.0, so the project needs to yank version 1.1.0-1, surely?

uranusjr commented

It seems that PyPI is showing the version incorrectly; the web page shows 1.1.0, but the listed file is actually 1.1.0-1.

I think the issue here is the new resolver is not excluding the yanked version correctly, but only try it last when all other choices are exhausted. This is inherited from the shared code that actually fetches packages, see #8262.

potiuk commented

It seems that PyPI is showing the version incorrectly; the web page shows 1.1.0, but the listed file is actually 1.1.0-1.

I think the issue here is the new resolver is not excluding the yanked version correctly, but only try it last when all other choices are exhausted. This is inherited from the shared code that actually fetches packages, see #8262.

@pfmoore -> exactly what @uranusjr wrote. This package https://pypi.org/project/oauthlib/1.1.0/ is correctly yanked. When you look at the download files, you will see that the actual file to download is 1.1.0-1.whl and tar.gz , which is exactly what the new resolver complains about. So the problem is in PIP. There are many other problems but here there are three very concrete problems here:

  • why those packages are at all downloaded if it in constraint file we have oauthlib==3.1.0 . I do not understand this and noone could provide any explanation yet.

  • why != does not work excluding it. Note @uranusjr @pfmoore that I excluded it in the top "setup.py" ! i literally added != 1.1.0 and even != 1.1.0-1 to the setup.py oauthlib - but it did not help

  • why yanking the release does not help ?

notatallshaw commented

Version 1.1.0-1 is different than 1.1.0, so the project needs to yank version 1.1.0-1, surely?

I know you must be very busy right now with lots of Pip issues but this is the problem being discussed on this issue. Oauthlib is listed as 1.1.0 on Pypi but then the pip resolver downloads it it reads it as 1.1.0-1 and errors out.

It has been been advised that for the Pip resolver team that is not a priority and to do one of the workarounds:

  • Yank the package - Doesn't work: Even though the package is yanked the pip resolver downloads, extracts it, and errors out
  • Block the version in the constraints file - Doesn't work: oathlib==3.1.0 was in the constraints file already, adding oauthlib!=1.1.0 and oauthlib!=1.1.0-1 produced the same error
  • Block the version in the top level requirements - Doesn't work: As @potiuk has listed, he has added oauthlib!=1.1.0 and oauthlib!=1.1.0-1 in the setup.py of airflow and still gets the same error

So there is a bug in the new pip resolver and all workarounds suggested have failed. Can you please consider increasing the priority of either this issue or the fact that none of the workarounds work as suggested.

pfmoore commented

Oauthlib is listed as 1.1.0 on Pypi but then the pip resolver downloads it it reads it as 1.1.0-1 and errors out.

So the file oauthlib-1.1.0-1.tar.gz on https://pypi.org/simple/oauthlib/ has a data-yanked="Wrong metadata published - preventing latest pip to work (#774)" attribute. Pip should ignore this, but doesn't. This is a bug (#8262), and will get fixed. We're not treating that issue as "not a priority", but we have extremely limited resources, and to be honest, we're pretty swamped right now trying to triage issues - many of which are taking an inordinate amount of time, and are quite stressful to discuss. In case it's not obvious, we'd be very grateful if someone came up with a PR for it, but otherwise we'll get to it as soon as we can.

What's lesser priority here is better handling of packages with incorrect metadata. We'd love to provide more informative errors in those cases, or deal with them more gracefully, but no matter what we choose, the metadata will still be incorrect, and we won't be able to do anything to avoid that. So anything pip does here will be a mitigation, not a fix, and we're focusing on fixes right now.

The oauthlib project has done their bit by yanking the invalid release. I hope they are also fixing their release processes so this doesn't happen again, but I'm happy if that's lower priority for them. It's now back to pip to address #8262. We're doing that, and you needn't be concerned about that issue having been de-prioritised (you may reasonably be concerned that a project like pip that's so critical to so many people, is critically under-resourced - but that's a longer term problem and not one we're going to solve this week 🙁)

There may be other workarounds. Some have been suggested, but haven't worked. OK, understood. Yanking remains the "correct" approach. I'm not entirely sure why the other suggestions aren't working, but it may be because the metadata is invalid (the old "garbage in, garbage out" principle) or it may be that there are edge cases that we don't fully understand involved. I'd assume you would prefer us to focus the time we have to spend on this issue on fixing #8262, and not on looking for yet more possible workarounds.

For what it's worth (not a lot!) #8262 isn't a "resolver issue", and probably exists in both the old and new resolvers. So it may well have not had much focus because we've been getting pressure to focus on issues related to the resolver. Now that we're aware of it, we have corrected that. But we still only have limited resources, so bear with us.

notatallshaw commented

@pfmoore Understood and thanks for your time but 2 minor corrections in your assessment of the situations:

We'd love to provide more informative errors in those cases

This isn't an issue of reporting, if the pip resolver skipped over packages with invalid metadata there wouldn't be any failure to resolve dependencies and install Airflow .

but it may be because the metadata is invalid (the old "garbage in, garbage out" principle) or it may be that there are edge cases that we don't fully understand involved.

All 3 workarounds listed (yanking package, constraints file, top level requirement) attempt to stop the garbage coming in. So the pip resolver shouldn't be downloading the package and reading the garbage metadata in the first place.

I actually would personally prefer the the other 2 workarounds be fixed so that it's possible to better control what packages the pip resolver chooses to download in the first place, suddenly having pip download dozens of packages it doesn't need to at install time is a separate OT issue. But I'm sure the Airflow project would be happy to just get to a state where it's possible to install using the new pip resolver.

potiuk commented

But I'm sure the Airflow project would be happy to just get to a state where it's possible to install using the new pip resolver.

Just to be frank, we can leave wit 2.20.4 for a while. We updated docs in our installation guides for upcoming 1.10.14rc1 and it will contain pre-requisite of downgrading to it or using the legacy flag. But will have to pay the price of explaining our users that they have to go through this additional hoop, and this might start taking the toll on the upcoming 2.0.0 release (initially planned next week) as we will have to respond to our users and if we know that fixes are coming soon, we'd rather delay the release by a few days maybe.

It's going to be release candidate of a first major release which we were working on for 2 years, so we prefer to do it right rather than - similarly as PIP - expose ourselves to "testing in production" and bear the cost of it.

So while not a disaster, it is definitely an inconvenience for us and if those problems are solved within a few days (and looking at the state and number of the problems I think it's unlikely) - we will bear it. But if this would last weeks, then probably temporary reverting to old resolver might be actually more "user friendly" approach for PIP maintainers (as I suggested in #9187 (comment) ).

uranusjr commented

I put together a patch to address the yanked issue, available in #9226, and I think you can install it with

pip install "pip @ https://github.com/uranusjr/pip/archive/new-resolver-skip-yanked-unless-only.zip"

Could both of you test whether it would indeed avoid the yanked version?

  • ❤️2
kaxil commented

I put together a patch to address the yanked issue, available in #9226, and I think you can install it with

pip install "pip @ https://github.com/uranusjr/pip/archive/new-resolver-skip-yanked-unless-only.zip"

Could both of you test whether it would indeed avoid the yanked version?

I can confirm this worked:

INFO: pip is looking at multiple versions of jsonpatch to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of oauthlib[signedtoken] to determine which version is compatible with other requirements. This could take a while.
Collecting oauthlib[signedtoken]>=1.0.0
  Downloading oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
     |████████████████████████████████| 143 kB 6.6 MB/s
  Downloading oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
     |████████████████████████████████| 142 kB 7.1 MB/s
  Downloading oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
     |████████████████████████████████| 142 kB 7.3 MB/s
  Downloading oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
     |████████████████████████████████| 124 kB 6.2 MB/s
  Downloading oauthlib-2.0.6.tar.gz (127 kB)
     |████████████████████████████████| 127 kB 7.4 MB/s
  Downloading oauthlib-2.0.5.tar.gz (129 kB)
     |████████████████████████████████| 129 kB 7.0 MB/s
  Downloading oauthlib-2.0.4.tar.gz (127 kB)
     |████████████████████████████████| 127 kB 5.5 MB/s
  Downloading oauthlib-2.0.3.tar.gz (127 kB)
     |████████████████████████████████| 127 kB 7.6 MB/s
  Downloading oauthlib-2.0.2.tar.gz (125 kB)
     |████████████████████████████████| 125 kB 7.0 MB/s
  Downloading oauthlib-2.0.1.tar.gz (122 kB)
     |████████████████████████████████| 122 kB 7.6 MB/s
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
     |████████████████████████████████| 122 kB 3.4 MB/s
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
     |████████████████████████████████| 111 kB 7.2 MB/s
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 6.3 MB/s
  Downloading oauthlib-1.0.3.tar.gz (109 kB)
     |████████████████████████████████| 109 kB 5.8 MB/s
  Downloading oauthlib-1.0.2.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 6.7 MB/s
  Downloading oauthlib-1.0.1.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 5.6 MB/s
  Downloading oauthlib-1.0.0.tar.gz (108 kB)
     |████████████████████████████████| 108 kB 5.8 MB/s

@pfmoore Sure let me cut you some slack -- But I can confirm that the fix works for "yank" issue.

FYI: I am also one of the maintainers of the Apache Airflow project so I can understand your pain and frustrations.

notatallshaw commented

I put together a patch to address the yanked issue, available in #9226, and I think you can install it with

pip install "pip @ https://github.com/uranusjr/pip/archive/new-resolver-skip-yanked-unless-only.zip"

Could both of you test whether it would indeed avoid the yanked version?

Thanks, testing, it does seem to resolve the yank issue. When going over the oauthlib requirements it did not fail at the usual step and I did not see oauthlib-1.1.0-1.tar.gz download.

Unfortunately I can't completely confirm success as there seems to be some dependency loop / extremely long graph walk involving setuptool, I have seen this on my screen now 518 times over 25 minutes so far:

Requirement already satisfied: setuptools>=34.0.0 in yank_resolved_test\lib\site-packages (from google-api-core<2dev,>=1.21.0->google-api-python-client<2.0.0,>=1.6.0->apache-airflow[all]) (50.3.2.post20201202)

I will let the resolver continue to run indefinitely and come back if I see the yank issue.

kaxil commented

Indeed. Same here, the "yanking" issue looks fixed with the patch but the installations are still going on. Maybe the same dependency loop like you are seeing @notatallshaw

pfmoore commented

I can understand your pain and frustrations.

Thanks, and my apologies to you. I let my frustration show. Deleted my comment too.

  • ❤️1
brainwane commented

Thanks @kaxil and @notatallshaw. To give a broad idea: once we have a verified fix and we feel confident about merging it, we can often make a point release within something like a business day. But please notice all the hedging since of course sometimes nasty CI problems come up, we realize that a particular set of fixes should go in together, etc.

(For reference: @pradyunsg and I are currently being paid to work on pip; the funding ran out to pay @pfmoore and @uranusjr so they're contributing right now as volunteers. Starting in January I won't be paid to work on pip, either.)

  • 👍1
kaxil commented

Thanks @kaxil and @notatallshaw. To give a broad idea: once we have a verified fix and we feel confident about merging it, we can often make a point release within something like a business day. But please notice all the hedging since of course sometimes nasty CI problems come up, we realize that a particular set of fixes should go in together, etc.

(For reference: @pradyunsg and I are currently being paid to work on pip; the funding ran out to pay @pfmoore and @uranusjr so they're contributing right now as volunteers. Starting in January I won't be paid to work on pip, either.)

Having worked as a volunteer for a long time myself on Airflow before working with Astronomer where I am being paid, I definitely understand it first hand. More power to you all who are working on PIP. We will try to help out as much as we can.

  • 👍1
ashb commented

Drop me an email @pfmoore and @uranusjr (when you have a spare momemt!) -- I'd like to see if my company sponsor you for a bit more!

  • 👍2
potiuk commented

@ashb - can you forward it to me as well? We can also help to reach out to other companies like Google or Amazon. They have stakes in Airflow project and if the problem is funding, we can try to - collectively - help. There are some open source funds in those companies and we can definitely help to reach our. Happy to help with that @pfmoore @uranusjr as well (I work with @ashb so just one e-mail to him is enough)..

notatallshaw commented

I will let the resolver continue to run indefinitely and come back if I see the yank issue.

Update on the test, after 6 hours pip exited with no error text and no package installation.

I am running the test again with verbose option to see if it's a reproducible error and if the verbose logging gives me any further information about why pip exited with no information.

sbidoul commented

We noticed a case where the use of post versions triggers the pip metadata consistency check:

$ pip install "docutils>=0.15,<0.15.2"
Collecting docutils<0.15.2,>=0.15
  Using cached docutils-0.15.1-post1.tar.gz (1.7 MB)
ERROR: Requested docutils<0.15.2,>=0.15 from https://files.pythonhosted.org/packages/d4/12/6c3fd74a590c7327c98cae008c11d536029fa9cd7924de477e8cb8804186/docutils-0.15.1-post1.tar.gz#sha256=f33ddb723332c6d6b6d99731ee1fc0c35eb4044a2df5cca1c64c8aa78eaf22cb has different version in metadata: '0.15.1'

Here the issue is that PyPI serves 0.15.1-post1 while the generated metadata says 0.15.1.

PEP 440 says "Some projects use post-releases to address minor errors in a final release that do not affect the distributed software (for example, correcting an error in the release notes).".

So does that mean that publishing a post release without changing the version metadata is allowed and pip should accept it ?

pfmoore commented

So does that mean that publishing a post release without changing the version metadata is allowed and pip should accept it ?

No. I'd read that as simply noting that post-releases are available for the purpose of non-functional changes like doc releases where you don't want to make a "full" version bump (and some projects use them for that). But it doesn't imply that the metadata can be inconsistent.

uranusjr commented

I would’ve said no a week ago, but PyPI seems to somewhat treat post releases kind of the same as their non-post counterpart. For example, docutils 0.15.1.post1 and 0.15.1 are listed on the same page, and [the title of that page says “0.15.1”(https://pypi.org/project/docutils/0.15.1/#files). But prereleases get their own version page.

So I think a case can be made either way, and a PyPA-wide discussion may be necessary to decide on the appropriate UX to this.

yahiaali commented

I'm running into the same issue as @potiuk. Is there a workaround that pip users can do for now while the maintainers work on a patch? For example, would reverting to an earlier version of pip resolve the error?

potiuk commented

I'm running into the same issue as @potiuk. Is there a workaround that pip users can do for now while the maintainers work on a patch? For example, would reverting to an earlier version of pip resolve the error?

Yep. you can use PIP 20.24 (pip install --upgrade 'pip==20.2.4') or use --use-deprecated legacy-resolver switch in pip 20.3

  • 👍1
edmorley commented

The fix for this (#9264) was reverted in #9293 - should this issue be reopened?