Audit binaries.pantsbuild.org find-links impact
Closed this issue · 2 comments
Currently the scie-pants Pants installs leverage our S3 bucket mounted at https://binaries.pantsbuild.org as an assured source of wheels for all platforms supported by Pants. This is critical to ensure the Python Build Standalone CPython distributions it uses are not called upon to compile native extension sdist-only wheels, which will not currently work. This works well but could be expensive in $ depending ho frequently wheels are pulled from the bucket - think CI. The factors to analyze:
- Given the version of Pip scie-pants uses to install Pants (we can control this), are the --find-links used as a fall-back only after trying PyPI 1st or not?
- What is the inventory of native wheels Pants uses, especially biased towards more recent / more heavily used versions and what is the breakdown of availability of these wheels per platform Pants supports.
The idea is to minimize S3 cost, ideally without impact to scie-pants usability across supported platforms and Pants versions. Since the installer logic is now in Python, it seems reasonable to tackle fairly complex logic if need be to turn on find links only for the platforms and Pants versions needed if need be.
One ~easy way to analyze this might be to create a Pex lock file for each version of Pants and then analyze those JSON documents.
Alright, adding a --log to our use of pip in install_pants.py and running the integration tests reveals 0 use of wheels from our find-links. The find-links are consulted, just never needed. This is on Linux where wheels are plentifully available; which is also our most likely CI target. But things tail off naturally. The second most likely CI target is macOS x86_64 since CI providers actually offer these machines, but, as a consequence, wheels tend to be available for these Macs. We only fall down on macOS arm, but that's exactly where we should fall down and need to use our s3 bucket; so the cost appears to be minimized already and just the pip version used to install needs to be pinned to ensure the current --find-links fall-back semantics.