pypi

sentry internal pypi

this repository contains the tools to import and/or build packages from public pypi for the platforms and achitectures required for sentry development.

adding packages

packages are configured in the packages.ini file.

the easiest way to add a package and its dependencies is to use:

python3 -m add_pkg PKGNAME

each section is an individual package and has some additional instructions which helps for building.

don't worry too much about the formatting, an auto-formatter will ensure the format is correct.

most packages won't need special build instructions and the section contents can be left blank:

[botocore==1.25.12]

[simplejson==3.17.2]
[simplejson==3.17.6]

Anyone in the Engineering team can approve Pull Requests, but it's preferred to get somebody from your team with context to approve your PR.

apt_requires / brew_requires

some packages require special system-level build dependencies, these can be configured using apt_requires (linux) and brew_requires (macos)

[xmlsec==1.3.12]
apt_requires =
    libxmlsec1-dev
    pkg-config
brew_requires =
    libxmlsec1
    pkg-config

custom_prebuild

sometimes the dependencies aren't packaged for apt / brew and you need a custom script to set them up. this script will be passed a single "prefix" directory (which contains the standard bin / lib / include / etc. structure).

the script should set up whatever tools are necessary inside only that directory

[google-crc32c==1.3.0]
custom_prebuild = prebuild/crc32c 1.1.2

likely_binary_ignore

when the sdist has source files of a compiled language (see BINARY_EXTS), we would expect compiled things (e.g. binary executables, shared objects) in the wheel. sometimes this isn't the case, and you can ignore those source files:

[salt==3006.6]
likely_binary_ignore =
    salt-3006.6/pkg/old/smartos/esky/sodium_grabber.c

python_versions

some packages are only intended for particular python versions (or don't otherwise build cleanly). the builds can be filtered using python_versions (though usually you should try and upgrade the relevant package).

[backports-zoneinfo==0.2.1]
python_versions = <3.9

validation

after building the packages will be checked that they can install and import

validate_extras

sometimes you may need to hint the validation tooling of additional requirements

an example is black which has a blackd top-level but requires an optional dependency to use (the black[d] extra). you can hint at this via validate_extras

[black==22.3.0]
validate_extras = d

validate_incorrect_missing_deps

sometimes packages incorrectly specify their dependencies. you can use this option to add import-time dependencies (though you should try and send a PR to fix those packages!)

one example is dictpath which depends on six but doesn't list it:

[dictpath==0.1.3]
validate_incorrect_missing_deps = six

validate_skip_modules

this should usually not be used but sometimes you need to skip importing some top-level modules due to side-effects or weird runtime requirements

[pyuwsgi==2.0.20]
validate_skip_imports = uwsgidecorators