pypi/conveyor

Provide content addressable URLs -- download by hash

Opened this issue · 6 comments

haampie commented

What's the problem this feature will solve?

In the Spack package manager we register the sha256 of the sources of any package, whether it's Python, C, C++, or Fortran.

For PyPI hosted packages we either have to

  1. make an API request to figure out the download URL
  2. store both the sha256 and the download URL
  3. make an educated guess about the download URL

Option number 3 is pain due to inconsistencies, e.g.

https://files.pythonhosted.org/packages/source/F/Fiona/Fiona-1.9.4.tar.gz
https://files.pythonhosted.org/packages/source/f/fiona/fiona-1.9.5.tar.gz  # inconsistent capitalization

or

https://pypi.org/packages/source/b/bitstring/bitstring-3.1.5.zip
https://pypi.org/packages/source/b/bitstring/bitstring-4.0.2.tar.gz  # inconsistent archive

Describe the solution you'd like

We'd prefer to only store the hash and do a single request to download the wheel / sdist from PyPI, without having to make a guess or deal with exceptions in naming.

That means we'd like to download by hash.

For example, if we wanna download black-24.2.0-py3-none-any.whl, which has a sha256 e8a6ae970537e67830776488bca52000eaa37fa63b9988e8c487458d9cd5ace6 it would be great if that was just one request to

https://files.pythonhosted.org/packages/black/sha256:e8a6ae970537e67830776488bca52000eaa37fa63b9988e8c487458d9cd5ace6

and have that redirect to the relevant download URL.

haampie commented

That is not a helpful way to deal with this issue. Can you please reopen?

The suggestion

def source_url(name, version):
    return f'{host}/packages/source/{name[0]}/{name}/{name}-{version}.tar.gz'

is wrong for both examples provided:

https://files.pythonhosted.org/packages/source/F/Fiona/Fiona-1.9.4.tar.gz # OK
https://files.pythonhosted.org/packages/source/f/fiona/fiona-1.9.4.tar.gz # 404 - not consistent with source_url
https://files.pythonhosted.org/packages/source/f/fiona/fiona-1.9.5.tar.gz # OK

https://pypi.org/packages/source/b/bitstring/bitstring-3.1.5.zip # OK
https://pypi.org/packages/source/b/bitstring/bitstring-3.1.5.tar.gz # 404 -- not consistent with source_url(...)
https://pypi.org/packages/source/b/bitstring/bitstring-4.0.2.tar.gz # OK
  1. Package name normalization does not help.
  2. Hard-coded .tar.gz is simply wrong.

It's not some historical artifact, the [fF]iona example is very recent, it looks like you're not applying normalization on the PyPI side, so either that has to be implemented, or the incorrect documentation you linked to should be removed.

The suggestion to allow download (normalize(name), hash) tuple avoids any naming issues.

adamjstewart commented

Note that these inconsistencies are even worse for wheels, where things may be py2.py3, cp312, or dozens of other version-, ABI-, and platform-specific tags.

  • 👍1
haampie commented

True, but those are expected to be known statically:

def wheel_url(name, version, build_tag, python_tag, abi_tag, platform_tag):

but I agree with the premise. In our package manager we want to use sdist and build binaries when packages are platform specific.

So, we wanna be able to download universal wheels or sdist.

For universal wheels we can do build_tag=None, abi_tag="none", platform_tag="any", but indeed python_tag would still have to be known / stored for each version in the package manager, as it can be py2.py3 or py3 -- that's annoying.

  • 👍1
ewdurbin commented

I created a sketch of what's requested in #31, but I am not wholly convinced of its usefulness.

Ultimately conveyor is a fairly low priority service for PyPI and we recommend the approach you identified:

  1. make an API request to figure out the download URL

As the correct long term solution.

konstin commented

We have a similar use case:

When installing from a lockfile, we need to know the url of each asset for performance and reliability reasons (we don't want to go through the index page again), its size (start the largest download first, progress bars) and its hash (for integrity). In our lockfile this looks as shown below. With the current pypi url structures this means repeating https://files.pythonhosted.org/packages/ for everything single distribution and storing two hex ids, e.g. for the source dist in the example one is the 63/09/c1bc53dab74b1816a00d8d030de5bf98f724c52c1635e07681d312f20be8 in the url and then the f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5 for the hash.

I'm looking for reducing the amount of strings we have to store in a lockfile. A smaller lockfile is easier to audit, faster to parse and makes better git diff. For the urls specifically, it would be nice lockfile to have a way to construct a direct url without serializing the entire url, possibly for pypi, ideally even cross index. Let's say e.g. we serialize file-prefix = "https://files.pythonhosted.org/disitribution/" and can then find every file at https://files.pythonhosted.org/disitribution/ + sha256 + / + distribution filename.

sdist = { url = "https://files.pythonhosted.org/packages/63/09/c1bc53dab74b1816a00d8d030de5bf98f724c52c1635e07681d312f20be8/charset-normalizer-3.3.2.tar.gz", hash = "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5", size = 104809 }
wheels = [
    { url = "https://files.pythonhosted.org/packages/d1/b2/fcedc8255ec42afee97f9e6f0145c734bbe104aac28300214593eb326f1d/charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:0b2b64d2bb6d3fb9112bafa732def486049e63de9618b5843bcdd081d8144cd8", size = 192892 },
    { url = "https://files.pythonhosted.org/packages/2e/7d/2259318c202f3d17f3fe6438149b3b9e706d1070fe3fcbb28049730bb25c/charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:ddbb2551d7e0102e7252db79ba445cdab71b26640817ab1e3e3648dad515003b", size = 122213 },
    { url = "https://files.pythonhosted.org/packages/3a/52/9f9d17c3b54dc238de384c4cb5a2ef0e27985b42a0e5cc8e8a31d918d48d/charset_normalizer-3.3.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:55086ee1064215781fff39a1af09518bc9255b50d6333f2e4c74ca09fac6a8f6", size = 119404 },
    { url = "https://files.pythonhosted.org/packages/99/b0/9c365f6d79a9f0f3c379ddb40a256a67aa69c59609608fe7feb6235896e1/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f4a014bc36d3c57402e2977dada34f9c12300af536839dc38c0beab8878f38a", size = 137275 },
    { url = "https://files.pythonhosted.org/packages/91/33/749df346e93d7a30cdcb90cbfdd41a06026317bfbfb62cd68307c1a3c543/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a10af20b82360ab00827f916a6058451b723b4e65030c5a18577c8b2de5b3389", size = 147518 },
    { url = "https://files.pythonhosted.org/packages/72/1a/641d5c9f59e6af4c7b53da463d07600a695b9824e20849cb6eea8a627761/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8d756e44e94489e49571086ef83b2bb8ce311e730092d2c34ca8f7d925cb20aa", size = 140182 },
    { url = "https://files.pythonhosted.org/packages/ee/fb/14d30eb4956408ee3ae09ad34299131fb383c47df355ddb428a7331cfa1e/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90d558489962fd4918143277a773316e56c72da56ec7aa3dc3dbbe20fdfed15b", size = 141869 },
    { url = "https://files.pythonhosted.org/packages/df/3e/a06b18788ca2eb6695c9b22325b6fde7dde0f1d1838b1792a0076f58fe9d/charset_normalizer-3.3.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6ac7ffc7ad6d040517be39eb591cac5ff87416c2537df6ba3cba3bae290c0fed", size = 144042 },
    { url = "https://files.pythonhosted.org/packages/45/59/3d27019d3b447a88fe7e7d004a1e04be220227760264cc41b405e863891b/charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:7ed9e526742851e8d5cc9e6cf41427dfc6068d4f5a3bb03659444b4cabf6bc26", size = 138275 },
    { url = "https://files.pythonhosted.org/packages/7b/ef/5eb105530b4da8ae37d506ccfa25057961b7b63d581def6f99165ea89c7e/charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:8bdb58ff7ba23002a4c5808d608e4e6c687175724f54a5dade5fa8c67b604e4d", size = 144819 },
    { url = "https://files.pythonhosted.org/packages/a2/51/e5023f937d7f307c948ed3e5c29c4b7a3e42ed2ee0b8cdf8f3a706089bf0/charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:6b3251890fff30ee142c44144871185dbe13b11bab478a88887a639655be1068", size = 149415 },
    { url = "https://files.pythonhosted.org/packages/24/9d/2e3ef673dfd5be0154b20363c5cdcc5606f35666544381bee15af3778239/charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_s390x.whl", hash = "sha256:b4a23f61ce87adf89be746c8a8974fe1c823c891d8f86eb218bb957c924bb143", size = 141212 },
    { url = "https://files.pythonhosted.org/packages/5b/ae/ce2c12fcac59cb3860b2e2d76dc405253a4475436b1861d95fe75bdea520/charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:efcb3f6676480691518c177e3b465bcddf57cea040302f9f4e6e191af91174d4", size = 142167 },
    { url = "https://files.pythonhosted.org/packages/ed/3a/a448bf035dce5da359daf9ae8a16b8a39623cc395a2ffb1620aa1bce62b0/charset_normalizer-3.3.2-cp312-cp312-win32.whl", hash = "sha256:d965bba47ddeec8cd560687584e88cf699fd28f192ceb452d1d7ee807c5597b7", size = 93041 },
    { url = "https://files.pythonhosted.org/packages/b6/7c/8debebb4f90174074b827c63242c23851bdf00a532489fba57fef3416e40/charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl", hash = "sha256:96b02a3dc4381e5494fad39be677abcb5e6634bf7b4fa83a6dd3112607547001", size = 100397 },
    { url = "https://files.pythonhosted.org/packages/28/76/e6222113b83e3622caa4bb41032d0b1bf785250607392e1b778aca0b8a7d/charset_normalizer-3.3.2-py3-none-any.whl", hash = "sha256:3e4d1f6587322d2788836a99c69062fbb091331ec940e02d12d179c1d53e25fc", size = 48543 },
]

If we could construct the url, we could slim this entry:

sdist = { filename = "charset-normalizer-3.3.2.tar.gz", hash = "sha256:f30c3cb33b24454a82faecaf01b19c18562b1e89558fb6c56de4d9118a032fd5", size = 104809 }
wheels = [
    { filename = "charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:0b2b64d2bb6d3fb9112bafa732def486049e63de9618b5843bcdd081d8144cd8", size = 192892 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:ddbb2551d7e0102e7252db79ba445cdab71b26640817ab1e3e3648dad515003b", size = 122213 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:55086ee1064215781fff39a1af09518bc9255b50d6333f2e4c74ca09fac6a8f6", size = 119404 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f4a014bc36d3c57402e2977dada34f9c12300af536839dc38c0beab8878f38a", size = 137275 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a10af20b82360ab00827f916a6058451b723b4e65030c5a18577c8b2de5b3389", size = 147518 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8d756e44e94489e49571086ef83b2bb8ce311e730092d2c34ca8f7d925cb20aa", size = 140182 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90d558489962fd4918143277a773316e56c72da56ec7aa3dc3dbbe20fdfed15b", size = 141869 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6ac7ffc7ad6d040517be39eb591cac5ff87416c2537df6ba3cba3bae290c0fed", size = 144042 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:7ed9e526742851e8d5cc9e6cf41427dfc6068d4f5a3bb03659444b4cabf6bc26", size = 138275 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:8bdb58ff7ba23002a4c5808d608e4e6c687175724f54a5dade5fa8c67b604e4d", size = 144819 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_ppc64le.whl", hash = "sha256:6b3251890fff30ee142c44144871185dbe13b11bab478a88887a639655be1068", size = 149415 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_s390x.whl", hash = "sha256:b4a23f61ce87adf89be746c8a8974fe1c823c891d8f86eb218bb957c924bb143", size = 141212 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:efcb3f6676480691518c177e3b465bcddf57cea040302f9f4e6e191af91174d4", size = 142167 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-win32.whl", hash = "sha256:d965bba47ddeec8cd560687584e88cf699fd28f192ceb452d1d7ee807c5597b7", size = 93041 },
    { filename = "charset_normalizer-3.3.2-cp312-cp312-win_amd64.whl", hash = "sha256:96b02a3dc4381e5494fad39be677abcb5e6634bf7b4fa83a6dd3112607547001", size = 100397 },
    { filename = "charset_normalizer-3.3.2-py3-none-any.whl", hash = "sha256:3e4d1f6587322d2788836a99c69062fbb091331ec940e02d12d179c1d53e25fc", size = 48543 },
]