Semver Config Resource

A Concourse resource for detecting desired semantic version changes and retrieving a set of semantic version-based configs, by one single YAML file.

Major features:

One yaml file to maintain a set of semantic version-based config items
Each config can have arbitrary structure where simply setting version_path to detect new versions and config_path to locate desired config elements
Support some useful semantic version detection patterns. For example, m.n.* means I care only Major and Minor version changes; while *.*.p means I care only Patch version changes

Maintainer

Bright Zheng

Major Change Logs

[2019-05-23] Initial release
[2019-07-16] Added support for detecting config changes within the same version
[2019-09-10] Released v1.0.0:
- Fixed the content hash detection issue by adding hash as part of the version identifiers
- Bumped yq to latest v2.4.0
[2019-09-24] Released v1.1.0:
- Added new version pattern "-" to cover more version-based scenarios, where the version might not be strictly semantic compatible
- Slightly refactored the code with more test cases

The Config File Example

The config file can be any sensible yaml format.

The simplest one might be:

# examples/version-only.yaml
product-1:
  version: "1.2.3"
product-2:
  version: "2.0.1"

So we have two products configured with different semantic versions, without any extra config items.

In real world, for example, while maintaining a PCF foundation with lots of products deployed, it can be something like this:

# examples/version-with-product-config.yaml
elastic-runtime:
  product-version: "2.3.4"
  pivnet-product-slug: elastic-runtime
  pivnet-api-token: ((pivnet_token))
  pivnet-file-glob: "*.pivotal"
  stemcell-iaas: google

pivotal-container-service:
  product-version: "1.4.0"
  pivnet-product-slug: pivotal-container-service
  pivnet-api-token: ((pivnet_token))
  pivnet-file-glob: "*.pivotal"
  stemcell-iaas: google

Where you have not only the semver, but also a set of config items.

Source Configuration

driver: Required. Currently only supports git
uri: Required. The location of the git repository
branch: Optional. The branch to track. Defaults to master if not set
private_key: Optional. The SSH private key that can be used for Git access authentication, if required.
username: Optional. The username that can be used for Git access authentication, if required.
password: Optional. The password that can be used for Git access authentication, together with username, if required.
config_file: Required. The relative path of the config file
config_path: Required. The yq-style path, e.g. x.y.z to locate the root of config items
initial_version: Optional. The initial version to start with. Defaults to 0.0.0 if not set
version_path: Required. The yq-style path, e.g. root.version to locate the root of semver item
version_pattern: Required. The pattern to be used for how to detect new versions. Refer to Version Patterns for the supported patterns

Behavior / Interfaces

`check`: Check for new versions based on desired Semver Pattern

The tracking git repository is cloned to temporary folder and check whether there is a desired newer version based on configured config_file, initial_version, version_path, version_pattern.

The extracted config elements will become resource's metadata, so please use integrated credential manager to handle credentials properly.

`in`: Get the config items of the desired version

The tracking git repository is cloned to temporary folder and extract the desired config elements based on config_file, config_path.

The extracted config elements will become resource's metadata and a file will be generated as the output which can be customized further by in parameters, as below.

Parameters

filename: Optional. The output file name, without file extension. Defaults to semver-config
format: Optional. Supported formats are: json and yaml. Defaults to yaml.

`out`: No-ops

No-ops for out.

Usage & Examples

Some pipeline examples are provided in /pipelines.

# pipelines/product-config.yaml
resource_types:
  - name: semver-config
    type: docker-image
    source:
      repository: itstarting/semver-config-concourse-resource
      tag: 1.1.0

resources:
- name: config
  type: semver-config
  source:
    driver: git
    uri: https://github.com/brightzheng100/semver-config-concourse-resource.git
    branch: master
    config_file: examples/version-with-product-config.yaml
    initial_version: 1.0.0                          # optional, defaults to 0.0.0 if not set
    config_path: "elastic-runtime"                  # [yq](https://github.com/mikefarah/yq)-style path
    version_path: "elastic-runtime.product-version" # [yq](https://github.com/mikefarah/yq)-style path
    version_pattern: "m.n.*"                        # refer to Version Patterns for how to detect new versions

jobs:
- name: job1
  plan:
  - get: config
    params:
      filename: my-lovely-config      # Optional. Defaults to `semver-config`
      format: yaml                    # Optional. Supported formats: `yaml`, `json`. Defaults to `yaml`
    trigger: true
  ...

Version Patterns

There are some version detection patterns supported:

m.*.*: ONLY Major (m) version change should be detected as a change, e.g. 2.*.* -> 3.*.*
m.n.*: BOTH Major (m) AND miNor (n) version change should be detected as a change, e.g. 2.*.* -> 3.*.*, 2.1.* -> 2.2.*
m.n.p: Any Major (m), miNor (n), and Patch (p) version change should be detected as a change, e.g. 2.*.* -> 3.*.*, 2.1.* -> 2.2.*, 2.1.5 -> 2.1.9
*.n.*: ONLY miNor (m) version change should be detected as a change, e.g. 2.2.* -> 2.3.*
*.*.p: ONLY Patch (p) version change should be detected as a change, e.g. 2.2.4 -> 2.2.9
-: This is a special version pattern, which means I'm just a version string which may not compatible with semantic versioning, but please detect my changes as well, based on the version string or content hash

Development

Prerequisites

yq: v2.3.0+ is tested; other versions may also work
jq: v1.5+ is tested; other versions may also work
Bats
Git client
Bash

Testing

Some submodules have been used in this repo for testing purposes.

$ mkdir -p tests/libs

$ git submodule add https://github.com/bats-core/bats-core.git tests/libs/bats
$ git submodule add https://github.com/ztombol/bats-support tests/libs/bats-support
$ git submodule add https://github.com/ztombol/bats-assert tests/libs/bats-assert

So if you want to run the test cases by yourself, do this:

$ git submodule update --init

$ bats tests/check.bats
$ bats tests/check-stemcell.bats
$ bats tests/in.bats
$ bats tests/in-stemcell.bats

Contributing

Please make all pull requests to the master branch and ensure tests are added to suitable /tests/*.bats file(s).

brightzheng100/semver-config-concourse-resource