speardrive: A Rust repository from da-x

speardrive - Dynamically create package repositories from CI job artifacts

A small web server that creates and serves read-only package repositories based on a plan from an accessed URL.

For example:

wget http://127.0.0.1:3200/myserver/foo/323/-/myserver/bar/111/-/rpm/repodata/repomd.xml

Immediately serve an RPM repo combining the RPM artifact outputs of job 323 of project foo and job 111 of project bar from configured Gitlab server myserver.

Highlights:

Supports Gitlab CI job artifacts.
Supports locally available artifacts.
Supports remotely available from URLs.
Supports generating RPM repositories.
Downloads artifacts and caches them locally per job.
Caches the combination of requested repositories.

Command line

speardrive --config-path <pathname>

URL format

Repos are created when their URLs are accessed, and the URLs define the read-only content of the repos.

<source-spec-a>/-/<source-spec-b>/-/.../<repo-type>

Where <source-spec> can be:

<gitlab-source-name>/<project-id>/<job-id>
<local-source-name>/<dirname>
<remote-static-name>/<dirname>

And <repo-type> can be:

rpm - Use createrepo_c to create local repositories

Configuration

Two types of sources are supported:

Gitlab job artifacts
A flat local directory

listen-addr: 127.0.0.1:3200
composites-cache: /storage/for/repo-composites
local-cache: /storage/for/cached-job-artifacts
gitlabs:
  'myserver':
     api-key: SomeAPIKEYObtainedFromGitlab
     hostname: git.myserver.com
local-source:
  local:
    root: /home/user/builds
remote-static:
  remote:
    base-url: https://some_static_site/suburl

Static remotes

For each <remote-static-name>/<dirname>, we will use the <base_url>/<dirname>/list.txt as the list of files to download under <base_url>/<dirname>. This list can be generated using find -type f.

Deployment example

Prebuilt images are available from dockerhub.

Example deployment script:

# Cleanup previous instance
docker rm -f speardrive 2>/dev/null

# Start new instance
docker run \
        --name speardrive \
        --network host \
        --user $(id -u):$(id -g) \
        --log-driver local --log-opt max-size=10m \
        -v /storage/speardrive:/storage/speardrive \
        -v $(pwd)/speardrive.yaml:/dist/config.yaml \
        -d "$@" \
        alonid/speardrive:0.1.6 /dist/speardrive \
        -c /dist/config.yaml  \
        serve

Example nginx location config

The following assumes a spawned speardrive instance that listens on 127.0.0.1:3200. You can serve it under a prefix using your existing nginx instance. For example:

location  /speardrive {
    rewrite /speardrive/(.*) /$1  break;
    proxy_pass         http://localhost:3200;
    proxy_redirect     off;
    proxy_set_header   Host $host;
}

Building from source

If you don't want to use the prebuilt docker image, you can build speardrive by yourself using Rust. With the Rust toolchain installed, run cargo install --path .

da-x/speardrive