wheelwright

This repo builds release wheels and source packages for Python libraries available as GitHub repositories. We're currently using it to build wheels for spaCy and our other libraries. The build repository integrates with Azure Pipelines and builds the artifacts for macOS, Linux and Windows on Python 3.6+. All wheels are available in the releases.

🙏 Special thanks to Nathaniel J. Smith for helping us out with this, to Matthew Brett for multibuild, and of course to the PyPA team for cibuildwheel and their hard work on Python packaging.

⚠️ This repo has been updated to use Azure Pipelines instead of Travis and Appveyor (see the v1 branch for the old version). We also dropped support for Python 2.7. The code is still experimental and currently mostly intended to build wheels for our projects. For more details on how it works, check out the FAQ below.

🎡 Usage

Quickstart

Fork or clone this repo and run pip install -r requirements.txt to install its requirements.
Generate a personal GitHub token with access to the repo, user and admin:repo_hook scopes and put it in a file github-secret-token.txt in the root of the repo. Commit the changes. Don't worry, the secrets file is excluded in the .gitignore.
Set up a GitHub service connection on Azure Pipelines with a personal access token and name it wheelwright. This will be used to upload the artifacts to the GitHub release.
Run python run.py build your-org/your-repo [commit/tag].
Once the build is complete, the artifacts will show up in the GitHub release wheelwright created for the build. They'll also be available as release artifacts in Azure Pipelines, so you can add a release process that uploads them to PyPi.

Package requirements

Wheelwright currently makes the following assumptions about the packages you're building and their repos:

The repo includes a requirements.txt that lists all dependencies for building and testing.
The project uses pytest for testing and tests are shipped inside the main package so they can be run from an installed wheel.
The package setup takes care of the whole setup and no other steps are required. build --sdist builds the sdist and cibuildwheel builds the wheels.

Setup and Installation

Make a local clone of this repo:

git clone https://github.com/explosion/wheelwright

Next, install its requirements (ideally in a virtual environment):

pip install -r requirements.txt

Click here to generate a personal GitHub token. Give it some memorable description, and check the box to give it the "repo" scope. This will give you some gibberish like f7d4d475c85ba2ae9557391279d1fc2368f95c38. Next go into your wheelwright checkout, and create a file called github-secret-token.txt and write the gibberish into this file.

Don't worry, github-secret-token.txt is listed in .gitignore, so it's difficult to accidentally commit it. Instead of adding the file, you can also provide the token via the GITHUB_SECRET_TOKEN environment variable.

Security notes

Be careful with this gibberish; anyone who gets it can impersonate you to GitHub.
If you're ever worried that your token has been compromised, you can delete it here, and then generate a new one.
This token is only used to access the wheelwright repository, so if you want to be extra-careful you could create a new GitHub user, grant them access to this repo only, and then use a token generated with that user's account.

Building wheels

Note that the run.py script requires Python 3.6+. If you want to build wheels for the v1.31.2 tag inside the explosion/cymem repository, then run:

cd wheelwright
python run.py build explosion/cymem v1.31.2

Eventually, if everything goes well, you'll end up with wheels attached to a new GitHub release and in Azure Pipelines. You can then either publish them via a custom release process, or download them manually:

python run.py download cymem-v1.31.2

In Azure Pipelines, the artifacts are available via the "Artifacts" button. You can also set up a release pipeline with twine authentication, so you can publish your package to PyPi in one click. Also see this blog post for an example.

🎛 API

`command` `run.py build`

Build wheels for a given repo and commit / tag.

python run.py build explosion/cymem v1.32.1

Argument	Type	Description
`repo`	positional	The repository to build, in `user/repo` format.
`commit`	positional	The commit to build.
`--package-name`	option	Optional alternative Python package name, if different from repo name.
`--universal`	flag	Build sdist and universal wheels (pure Python with no compiled extensions). If enabled, no platform-specific wheels will be built.
`--llvm`	flag	Build requires LLVM to be installed, which will trigger an additional step in Windows build pipeline.
`--rust`	flag	Build request Rust to be installed, which will trigger an additional step in Windows build pipeline. (Rust is installed by default in all other pipelines.)
`--skip-tests`	flag	Don't run tests (e.g. if package doesn't have any). Only supported for `--universal` builds.

`command` `run.py download`

Download existing wheels for a release ID (name of build repo tag). The downloaded wheels will be placed in a directory wheels.

python run.py download cymem-v1.31.2

Argument	Type	Description
`release-id`	positional	Name of the release to download.

Environment variables

Name	Description	Default
`WHEELWRIGHT_ROOT`	Root directory of the build repo.	Same directory as `run.py`.
`WHEELWRIGHT_WHEELS_DIR`	Directory for downloaded wheels.	`/wheels` in root directory.
`WHEELWRIGHT_REPO`	Build repository in `user/repo` format.	Automatically read from `git config`.
`GITHUB_SECRET_TOKEN`	Personal GitHub access token, if not provided via `github-secret-token.txt`.	-

⁉️ FAQ

What does this actually do?

The build command uses the GitHub API to create a GitHub release in this repo, called something like cymem-v1.31.2. Don't be confused: this is not a real release! We're just abusing GitHub releases to have a temporary place to collect the wheel files as we build them. Then it creates a new branch of this repo, and in the branch it creates a file called build-spec.json describing which project and commit you want to build.

When Azure Pipelines sees this branch, it springs into action, and starts build jobs running on a variety of architectures and Python versions. These build jobs read the build-spec.json file, and then check out the specified project/revision, build it, test it, and finally attach the resulting wheel to the GitHub release we created earlier.

What if something goes wrong?

If the build fails, you'll see the failures in the Azure Pipelines build logs. All artifacts that have completed will still be available to download from the GitHub release.

If you resubmit a build, then run.py will notice and give it a unique build ID – so if you run run.py build explosion/cymem v1.31.2 twice, the first time it'll use the id cymem-v1.31.2, and the second time it will be cymem-v1.31.2-2, etc. This doesn't affect the generated wheels in any way; it's just to make sure we don't get mixed up between the two builds.

As a package maintainer, what do I need to know about the build process?

Wheels are built using cibuildwheel. For native linux aarch64 builds, we use ec2buildwheel to run cibuildwheel on an EC2 instance.

Our projects use a single requirements.txt that includes both the build and test requirements. You could imagine splitting those into two separate files, in order to make sure that dependency resolution is working, that we don't have any run-time dependency on Cython, etc., but currently we don't.

We assume that projects use pytest for testing, and that they ship their tests inside their main package, so that you can run the tests directly from an installed wheel without access to a source checkout.

For simplicity, we assume that the repository name (in the clone URL) is the same as the Python import name (in the pytest command). You can override this on a case-by-case basis passing --package ... to the build command, but of course doing this every time is going to be annoying.

Aside from modifying the package setup, there isn't currently any way for a specific project to further customize the build, e.g. if they need to build some dependency like libblis that's not available on PyPI.

I'm not Explosion, but I want to use this too!

Currently we'd recommend using cibuildwheel instead for most use cases, but wheelwright is under the MIT license, so feel free if it makes sense for your project!

explosion/wheelwright