PackPack

PackPack is a simple tool to build RPM and Debian packages from git repositories:

Fast reproducible builds using Docker containers
Semantic versioning based on annotated git tags
Support for all major Linux distributions as targets

PackPack works best with GitHub, Travis CI and PackageCloud:

Push your code to GitHub
Build packages using Travis CI
Host repositories on PackageCloud

Motivation

PackPack is designed by folks from Mail.Ru Group, a leading Internet company in Europe, to automate release management cycle of open source products as well as of proprietary software.

Tarantool, an open-source general-purpose database and an application server, has dozens of git commits per day and this number is constantly increasing month after month. In order to deliver the best user experience and offer enterprise-level support quality, the Tarantool team packages almost every git commit from four git branches for (!) fifteen various Linux distribution.

Traditional tools, like mock and pbuilder, were tremendously slow and ridiculously overcomplicated. Customers had to wait hours for hotfix packages and the project paid thousands of dollars annually for hardware and electricy bills. Such cost are unacceptable for the most "free as in speech" open-source projects.

PackPack has reduced push-to-package time from hours to minutes. Tarantool team were even able to package all complementary modules, addons and connectors using this tool. Tarantool users now can also package their own proprietary modules in the same manner as official packages.

Our Users

Tarantool - general-purpose database and Lua application server.
ZoneMinder - full-featured, open source, state-of-the-art video surveillance software system.
SysBench - scriptable database and system performance benchmark.
IronSSH - secure end-to-end file transfer software developed by IronCore Labs.
MINC Toolkit V2 - Medical Imaging NetCDF Toolkit developed by McConnell Brain Imaging Centre.
LuaFun - functional programming library for Lua.
MsgPuck - simple and efficient MsgPack binary serialization library.
Phalcon - high performance PHP Framework.
MyHTML - a fast HTML Parser implemented as a pure C99 library.

Of course, PackPack itself is packaged using PackPack.

Supported Platforms

Distributions:

Debian Wheezy / Jessie / Stretch / Sid
Ubuntu Precise / Trusty / Xenial / Yakkety / Zesty
Fedora 24 / 25 / Rawhide
CentOS 6 / 7
Alpine (initial support)

Archictectures:

i386
x86_64
armhf (32-bit ARM with hardware floating-point)
aarch64 (64-bit ARM)

The actual list of distribution is available on Docker Hub. Please file an issue if you want more.

Getting Started

Install git, docker and any posix-compatible shell (bash, dash, zsh, etc.). The complicated one is Docker, please see the detailed guide on docs.docker.com web-site.
Add RPM spec to rpm/ folder of your git repository. The best way to create a new spec file for your product is to find an existing one for a similar software, rename and then modify it. See Fedora Git and Fedora Packaging Guidelines for details. Some examples are available from tarantool/modulekit repository.
Add debian/ folder to your git repository, as usual. Debian has complicated package structure and we strongly recommend to find a similar package in the official repositories, download it using apt-get source package command, copy and paste and then modify debian/ directory. Some examples are available from tarantool/modulekit repository.
Create an annotated major.minor git tag in your repository. PackPack will automatically set patch level based on the commit number from this tag in order to provide major.minor.patch semantic versioning:

$ git tag -a 1.0
$ git describe --always --long
1.0-0-g5c26e8b # major.minor-patch = 1.0-0
$ git push origin 1.0:1.0 # Push to GitHub

Clone PackPack repository:

myproject$ git clone https://github.com/packpack/packpack.git packpack

Try to build some packages for, say, Fedora 24. For the first time, Docker will download images from Docker Hub, please wait a little bit.

myproject$ OS=fedora DIST=24 ./packpack/packpack

The build artifacts will be stored into build/ directory:

myproject$ ls -1s build/
total 112
76 myproject-1.0.2-0.fc24.src.rpm
36 myproject-devel-1.0.2-0.fc24.x86_64.rpm

Of course, PackPack can also be installed from DEB/RPM packages:

# For Debian, Ubuntu and other Debian-based
curl -s https://packagecloud.io/install/repositories/packpack/packpack/script.deb.sh | sudo bash
# For Fedora, RedHat, CentOS and other RPM-based
curl -s https://packagecloud.io/install/repositories/packpack/packpack/script.rpm.sh | sudo bash

See PackPack Repositories for additional instructions.

How it Works

PackPack performs the following steps:

A Docker container is started using packpack/packpack:$OS$DIST image.
The source repository is mounted to the container as a read-only volume.
major.minor.patch version is extracted from git describe output.
A source tarball (product-major.minor.patch.tar.gz) is packed from files added to git repository.
For RPM package:
- spec file is copied from rpm/, Version: tag is updated according to extracted major.minor.patch version, %prep is updated to match the source tarball file name.
- A source RPM (product-major.minor.patch-release.dist.src.rpm) is built from the source tarball using generated spec file.
- BuildRequires are installed using dnf builddep or yum-builddep. Docker images already have a lot of packages pre-installed to speed up the build.
- rpmbuild is started to build RPM packages from the source RPM.
For Debian packages:
- debian/changelog is bumped with extracted major.minor.patch git version.
- Build-Depends are installed using mk-build-deps tool. Docker images already have a lot of packages pre-installed to speed up the build.
- A symlink for orig.tar.gz is created to the source tarball
- dpkg-buildpackage is started to build Debian packages.
Resulted packages, tarballs and log files are moved to /build volume, which is mounted by default to ./build directory of your git repository.

GitHub, Travis CI and PackageCloud

PackPack is designed to use with GitHub, Travis CI and PackageCloud.

Register free PackageCloud account and create a repository.
Add your GitHub project to Travis CI.
Add the following environment variables to the project settings on Travis CI:
- PACKAGECLOUD_TOKEN=<token> (secret)
- PACKAGECLOUD_USER=<username> (public)
- PACKAGECLOUD_REPO=<reponame> (public)
Click to see how.
Enable PackPack magic in .travis.yml file:

sudo: required
services:
  - docker

cache:
    directories:
     - $HOME/.cache

language: C

env:
    matrix:
      - OS=el DIST=6
      - OS=el DIST=7
      - OS=fedora DIST=24
      - OS=fedora DIST=25
      - OS=ubuntu DIST=trusty
      - OS=ubuntu DIST=precise
      - OS=ubuntu DIST=xenial
      - OS=ubuntu DIST=yakkety
      - OS=debian DIST=jessie
      - OS=debian DIST=wheezy
      - OS=debian DIST=stretch
      - OS=ubuntu DIST=xenial ARCH=i386
      - OS=debian DIST=jessie ARCH=i386

script:
 - git submodule update --init --recursive
 - git describe --long
 - git clone https://github.com/packpack/packpack.git packpack
 - packpack/packpack

deploy:
  # Deploy packages to PackageCloud
  provider: packagecloud
  username: ${PACKAGECLOUD_USER}
  repository: ${PACKAGECLOUD_REPO}
  token: ${PACKAGECLOUD_TOKEN}
  dist: ${OS}/${DIST}
  package_glob: build/*.{deb,rpm}
  skip_cleanup: true
  on:
    branch: master
    condition: -n "${OS}" && -n "${DIST}" && -n "${PACKAGECLOUD_TOKEN}"

Push changes to GitHub repository to trigger Travis CI build.
Check Travis CI logs and fix packaging problems, if any. Click to see how.
Get packages on your PackageCloud repository. Click to see how.
???
Star this project on GitHub if you like this idea.
PROFIT

That's it.

BTW, Travis CI allow to exclude some builds from matrix, see an example in Tarantool GitHub repo.

Configuration

PackPack can be configured via environment variables:

OS - target operating system name, e.g. fedora or ubuntu
DIST - target distribution name, e.g 24 or xenial
ARCH - target architecture, like on Docker Hub:
- i386
- x86_64
- armhf
- aarch64 It is possible to use ARCH=i386 on x86_64 host and ARCH=armhf on aarch64 host, but there is no way to run ARM images on Intel and vice versa. Docker is qemu and can't emulate foreign instruction set.
BUILDDIR - a directory used to store intermediate files and resulted packages (default is ./build).
PRODUCT - the name of software product, used for source tarball and source package, e.g. tarantool
VERSION - semantic version of the software, e.g. 2.4.35 (default is extracted for git describe).
RELEASE - the number of times this version of the software has been packaged (default is 1).
ABBREV - abbreviation of build metadata (default is git hash, extracted from git describe).
TARBALL_COMPRESSOR - a compression algorithm to use, e.g. gz, bz2, xz (default is xz).
CHANGELOG_NAME, CHANGELOG_EMAIL, CHANGELOG_TEXT - information used to bump version in changelog files.
DOCKER_REPO - a Docker repository to use (default is packpack/packpack).
CCACHE* - Config variables for ccache, such as CCACHE_DISABLE

See the full list of available options and detailed configuration guide in pack/config.mk configuration file.

The actual list of distribution is available on [Docker Hub] (https://hub.docker.com/r/packpack/packpack/tags/).

Contribution

PackPack is written on Makefiles and contains less than 300 lines of code. We've tried different variants, like Python, but GNU Make is actually the simplest (and fastest) one.

Any pull requests are welcome.

Please feel free to fork this repository for experiments. You may need to create your own repository on Docker Hub. Click to see how.

connortechnology/packpack