chacra: A Python repository from joehandzik

chacra

A REST API for storing and retrieving specific versions and architectures of (binary) packages.

The URL structure is very simple and allows to be explicit about what the type and version of a given binary package the API is providing:

/bianries/{package_name}/{version}/{distribution}/{distro_release}/{architecture}/$binary

So for a CentOS 7 x86_64 package for Ceph the url could look like:

/binaries/ceph/firefly/centos/7/x86_64/ceph-0.87.2-0.el7.centos.x86_64.rpm

Configuration

The service needs to be configured with a few items to correctly work with binaries.

binary_root

The binary_root is a required configuration item, it defines where binaries live so that when a new binary is POSTed the service will use this path to save the binary to.

distributions_root

The distributions_root is a required configuration item, it defines where the project specific distributions files will be stored when creating debian repositories.

credentials

The POST and DELETE HTTP methods are protected by default using basic HTTP authentication. The credentials must be defined in the configuration file for the service as follows:

api_user = 'username'
api_key = 'secret'

Self-discovery

The API provides informational JSON at every step of the URL about what is available as a resource. The following examples show informational output that can be consumed to dig deeper into the URL structure:

GET /binaries/:

{
    "ceph": ["firefly", "giant", "hammer"],
    "ceph-deploy": ["master"]
}

GET /binaries/ceph/:

{
    "firefly": ["centos", "redhat", "debian", "ubuntu"]
}

GET /binaries/ceph/firefly/:

{
    "centos": ["7", "6"],
    "debian": ["wheezy"],
    "ubuntu": ["trusty"],
}

GET /binaries/ceph/firefly/centos/:

{
    "7": ["x86_64"],
    "6": ["x86_64"]
}

GET /binaries/ceph/firefly/centos/7/:

{
    "x86_64": ["ceph-0.87.2-0.el7.centos.x86_64.rpm"]
}

GET /binaries/ceph/firefly/centos/7/x86_64/:

{
    "ceph-0.87.2-0.el7.centos.x86_64.rpm": {
        "signed": True,
        "size": "12M",
        "last_updated": "14 hours, 39 minutes, 19 seconds ago",
        "built-by": "ktdreyer",
    },
    "ceph-0.87.1-0.el7.centos.x86_64.rpm": {
        "signed": True,
        "size": "12M",
        "last_updated": "20 days, 6 hours, 39 minutes, 19 seconds ago",
        "built-by": "ktdreyer",
    },
    "ceph-0.87-0.el7.centos.x86_64.rpm": {
        "signed": True,
        "size": "12M",
        "last_updated": "35 days, 6 hours, 39 minutes, 19 seconds ago",
        "built-by": "alfredodeza",
    }

}

At this point, the keys for the JSON object represent the available binaries for the next part of the URL.

So another GET operation on that final URL would retrieve the actual binary. Other metadata may be provided, but the rule would be that all top-level keys are meant to be part of the consumable url.

Querying binary information

The search endpoint is /search/ and accepts a few keyword arguments. At the moment only simple querying is allowed (only matches are used) and no other logical operators can be used, like >= for example.

In its most simple form a query would look like:

/search/?name=ceph

Search terms don't need to be unique and successful responses will return an array of items found along with metadata about locations.

The supported query parameters are:

distro
distro_version
arch
ref
built_by
size
name

HTTP Responses:

200: Success. Body:

[
  {
    "ceph-0.87.2-0.el10.centos.x86_64.rpm": {
        "url": "/binaries/ceph/firefly/centos/10/x86_64/ceph-0.87.2-0.el10.centos.x86_64.rpm"
    }
  },
    "ceph-0.87.1-0.el10.centos.x86_64.rpm": {
        "url": "/binaries/ceph/firefly/centos/10/x86_64/ceph-0.87.1-0.el10.centos.x86_64.rpm"
    },
  }
]

HTTP Responses:

200: Success.

400: Invalid request. Body:

{ "msg": "invalid query params: {keys}" }

File resources

POST requests will create new resources when using the full url with all the metadata parts including the filename when uploading files. For example:

curl -F "file=@/home/user/repos/ceph-0.87.2-0.el10.centos.x86_64.rpm" https://chacra.ceph.com/binaries/ceph/firefly/centos/10/x86_64/

Note how a trailing slash is required as well as the full name of the binary.

If the binary exists it will not get overwritten unless the force value is set. Otherwise a 400 is returned.

If the force flag is set and the binary is overwritten a 200 is returned. If the resource does not exist, a 201 is returned.

File uploads cannot create metadata other than the file path where the binary is stored at.

User must configure the location of binary uploads in the config file, for a location relative to where the config file lives:

binary_root = '%(confdir)s/public'

Or any other absolute path is allowed too:

binary_root = '/opt/binaries'

Directory paths will follow the same structure as in URLs. For example, with a binary_root key that points to /opt/binaries/ the final location for a resource that lives in /binaries/ceph/firefly/centos/10/x86_64/ceph-0.87.1-0.el10.centos.x86_64.rpm/ would be /opt/binaries/ceph/firefly/centos/10/x86_64/ceph-0.87.1-0.el10.centos.x86_64.rpm/

HTTP Responses:

200: Resource was updated
201: Resource was created

400: Invalid request. Body:

{ "msg": "resource already exists and 'force' flag was not set" }

POST will create new items at given parts of the URL. For example, to create a new package, a POST to /binaries/ with an HTTP body that should look like:

{ "name": "my_new_package" }

For other parts of the URL the "name" key is also required.

HTTP Responses:

200: Success.

400: Invalid request. Body:

{ "msg": "my_new_package already exists" }

Binary metadata updates

For non-existing URLs a retroactive feature will create the rest of the url structure. For example, a new distribution release for CentOS 10 that didn't exist before at this url and for the following package:

/binaries/ceph/firefly/centos/10/x86_64/ceph-0.87.2-0.el10.centos.x86_64.rpm

Would create all the parts that didn't exist before ('10','x86_64', and 'ceph-0.87.2-0.el10.centos.x86_64.rpm' from our previous examples). This would happen with file uploads too.

The body for the POST HTTP request would still require the "name" key:

{ "name": "ceph-0.87.2-0.el10.centos.x86_64.rpm" }

Optional (but recommended key) is the built-by:

{
    "name": "ceph-0.87.2-0.el10.centos.x86_64.rpm",
    "built-by": "alfredodeza"
}

These requests need to go to the parent url part, so for the example above the HTTP request would go to:

/binaries/ceph/firefly/centos/10/x86_64/

Note the need for a trailing slash.

Force a rewrite of a binary metadata

If a POST is done to a binary URL that already exists, the API will return a 400 with a message indicating that the binary is already there.

But sometimes, rewriting a binary is needed and the API allows that with a flag in the JSON object when doing a POST:

{
    "name": "ceph-0.87.2-0.el10.centos.x86_64.rpm",
    "force": True
}

Again, note that this POST would need to go to the root of the url, following the examples above that would mean:

/binaries/ceph/firefly/centos/10/x86_64/

Note the need for a trailing slash.

HTTP Responses:

200: Success.

400: Invalid request. Body:

{ "msg": "resource already exists and 'force' flag was not set" }

Automatic Repositories

This service provides automatic repository creation per distribution version, with optional handling of combined repositories in the case of DEB packages.

The default repository structure URL looks like:

/repos/{project}/{ref}/{distro}/{distro version}/{REPO}

Defining custom repositories

To create repos that combine multiple distro versions, define them in repos dictionary in your config. Repos can only be created per REF, although configuration is done at PROJECT level, affecting all REFs. For example in config.py:

repos = {
    'ceph': {
        'combined': ['wheezy', 'precise', 'jessie'],
    }
}

The above configuration would create a "combined" repository of the defined versions. The repository would then be available at:

/repos/{project}/{ref}/combined/{combined REPO}

All other repos built for other other distro versions will still be available at the default endpoint:

/repos/{project}/{ref}/{distro}/{distro version}/{REPO}

Note

Creating a repository that combines multiple distro versions is only available for debian based distros.

Defining extra packages

For extra packages that may be coming from other projects, the configuration structure will allow for definition of them. For example, 'ceph-deploy' exists publicly in the 'ceph' repositories, just like 'radosgw-agent'. This inclusion would be defined at the project level, like:

repos = {
    'ceph': {
        'all': {
            'ceph-deploy': ['all'],
        },
        'firefly' {
            'radosgw-agent': ['all'],
            'ceph-deploy': ['v1.0.0', 'v1.1.1'],
        },
        'hammer' {
            'radosgw-agent': ['all'],
        },
        'giant' {
            'radosgw-agent': ['all'],
        },
        'combined': ['wheezy', 'precise', 'jessie'],
    }
}

The extras key would require those projects to be present in the chacra instance that is creating the repositories. # TODO: Maybe allow for URLs as well? That way packages could come from another source?

Disabling per project

Automatic repository creation can be disabled (on by default) in the configuration for repos. In the case of a project like ceph-deploy that is usually included in other repos, it could be disabled like:

repos = {
    'ceph-deploy': {
        'disabled': True
    }
}

Configuring distributions

Creating a debian repository requires a distributions file be created. Chacra will create these for each project by using the following configuration:

distributions = {
   "defaults": {
        "DebIndices": "Packages Release . .gz .bz2",
        "DscIndices": "Sources Release .gz .bz2",
        "Contents": ".gz .bz2",
        "Origin": "RedHat",
        "Description": "",
        "Architectures": "amd64 armhf i386 source",
        "Suite": "stable",
        "Components": "main",
    },
    "ceph": {
        "Description": "Ceph distributed file system",
    },
}

The defaults key is used for any project that doesn't have it's own explicitly defined key. This key isn't required, but it can be usueful when you have many projects with similar values in their distributions files.

If you want to add keys or modify keys that exist in defaults for a specific project, add that project name as a key of distributions and define the keys you'd need to override or add there.

Authentication

If authentication is configured, you can use the following flags to curl:

curl --basic -u myuser -k -F "file=@ceph-deploy-1.5.28-0.noarch.rpm" https://chacra.example.com/binaries/ceph/test/centos/10/x86_64/

You should also investigate https://pypi.python.org/pypi/chacractl, a client that wraps the chacra API and handles authentication in a configuration file, etc.

about the name

chakra is a quechua word to refer to a small farm in the outskirts, dedicated to produce food for the city.

Reference: https://en.wikipedia.org/wiki/Quechua

joehandzik/chacra

chacra

Configuration

binary_root

distributions_root

credentials

Self-discovery

Querying binary information

File resources

Binary metadata updates

Force a rewrite of a binary metadata

Automatic Repositories

Defining custom repositories

Defining extra packages

Disabling per project

Configuring distributions

Authentication

about the name