pypa/pip

pip 21 changes preference order of found packages

mboisson opened this issue · 4 comments

mboisson commented

Description

Issue first described in #9958 which was closed very quickly.

With version 21, pip's --find-links option is basically useless, because it will prefer packages found in the index. The whole point of having locally built wheels is to have them preferred over the ones available on pipy.

How can we obtain the old behavior of preferring local wheels back ? Is there a configuration parameter to revert this breaking change ?

Using --no-index is not a fix, because it blocks all wheels from the index. We don't want to block everything we don't have locally, we just want it to prefer what we have locally if we do. We need a --prefer-local or something like that.

Expected behavior

Not break previous behavior.

pip version

21.x

Python version

3.x

OS

CentOS

How to Reproduce

Use --find-links to provide a local directory of wheels to install both packages that you have locally and packages that are on pipy.

Output

No response

Code of Conduct

mboisson commented

This has also been a recurring issue in the past, but this new change is just making it worst. #6023

mboisson commented

Will this be addressed, or to we need to fork pip to fix this ?

uranusjr commented

#8606 may provide some context (--find-links is basically --extra-index-url with a different format).

Since pip does not sort packages by their sources, but the versions, it is established as an explicit policy to not prefer private package sources, as an effort to prevent users from trying to oerride package versions on a public index, since this is extremely brittle and expose you to an entire vector of supply chain attacks. While forking would get the old behaviour back, you will still be subject to those same vulnarabilities—unless you managed to fix the underlying supply chain issue, in which case please submit the patch back upstream! We recommend you to try one or more of the following approaches instead:

  1. Use direct URL references. Newer versions of pip have improved support around this functionality, and by specifying where you want a dependency explicitly, you are guaranteed to get it.
  2. Use local version identifiers in your locally-built wheels, and pin all requirements with == or ===. This ensures your wheels always have higher priority over the public index since PyPI.org disallows publishing distributions with the local version segment.
  3. Use --no-index or override the default index with --index-url, maybe with a pass-through index server to route individual packages to their “real” source. This is the best solution for teams with moderate size and up, since it give the devops complete control over what is installed from where, while minimisling mental overhead for other developers.

Additionally, we recommend enabling hash-checking to ensure you get what you expect. For approaches 1 and 2, you may also find --constraint useful if you have many private package builds.

mboisson commented

This is really annoying! PyPI is fine for some packages, but many packages provided in PyPI DO NOT WORK on clusters. If we do provide extra index URL, it is PRECISELY to avoid using those from PyPI.

I guess we will keep with version 20.x of pip until we have no choice and then we may fork pip.