danieltanfh95/mkdkr

Make + Docker + Shell = CI Pipeline

mkdkr

mkdkr = Makefile + Docker

Super small and powerful framework for build CI pipeline, scripted with Makefile and isolated with docker.

• Dependencies: [ make, docker, bash, git ]
• Two files only (Makefile and .mkdkr), less garbage on your repo
• All power of make, docker and bash
• Shipping and switch among CI engines like Circle CI, GitHub Actions, Gitlab-ci, Jenkins, Travis.. and more #using exporter
• Clean and elegant code syntax

Table of contents

• Usage
  • Installation
  • Create Makefile
  • Execute
  • Result
  • Export
• Reason
• Functions
  • @$(dkr)
  • instance:
  • service:
  • dind:
  • run:
• Includes
  • Explicit
  • Implicit
  • mkdkr.csv
  • Collection
• Examples
  • Simple
  • Service
  • DIND
  • Escapes
  • Shell
  • Sdtout
  • Pipelines
• Environment Variables
• Migration

Usage

Installation

# Download .mkdkr
curl https://raw.githubusercontent.com/rosineygp/mkdkr/master/.mkdkr > .mkdkr

# not required, but can be used as template
curl https://raw.githubusercontent.com/rosineygp/mkdkr/master/examples/simple.mk > Makefile

Makefile

Create a file with name Makefile and paste the following content

# Required header
include $(shell bash .mkdkr init)

job:	                          # job name
	@$(dkr)                       # required: load mkdkr (docker layer)
	instance: alpine              # create a docker container using alpine image
	run: echo "hello mkdkr!"      # execute a command inside container

Execute

# execute
make job

Result

start: job


instance: alpine 
20498831fe05f5d33852313a55be42efd88b1fb38b463c686dbb0f2a735df45c

run: echo hello mkdkr!
hello mkdkr!

cleanup:
20498831fe05

completed:
0m0.007s 0m0.000s
0m0.228s 0m0.179s

Export

Run your current Makefile in another engine, like travis or github actions, use the dynamic include exporter.

Reason

Build pipeline for a dedicated platform can take a lot of time to learn and test, with mkdkr you can test all things locally and run it in any pipeline engine like Jenkins, Actions, Gitlab-ci and others.

Functions

@$(dkr)

Load docker layer for mkdkr, use inside a target of Makefile.

shell-only:
	echo "my local shell job"

mkdkr-job:
	@$(dkr)            # load all deps of mkdkr
	intance: alpine
	run: echo "my mkdkr job"

instance:

Create a docker container, without special privileges.

my-instance:
	@$(dkr)
	instance: ubuntu:20.04     # create a instance

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

Calling instance: twice, it will replace the last container.

service:

Create a docker container in detached mode. Useful to bring up a required service for a job, like a webserver or a database.

my-service:
	@$(dkr)
	service: nginx    # up a nginx
	instance: alpine

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

instance or dind created after a service, will be automatically linked.

dind:

Create a docker instance with daemon access. Useful to build docker images.

my-dind:
	@$(dkr)
	dind: docker:19
	run: docker build -t my/dind .

Parameters:

• String, DOCKER_IMAGE *: any docker image name
• String|Array, ARGS: additional docker init args like (--cpus 1 --memory 64MB)

Return:

• String, Container Id

run:

Execute a command inside docker container [instance: or dind:] (the last one).

Is not possible to execute commands in a service.

Parameters:

• String|Array, command: any sh command eg. 'apk add nodejs'

Return:

• String, Command(s) output

Usage

my-run:
	@$(dkr)
	instance: alpine
	# run a command inside container
	run: apk add curl

	instance: debian
	# avoid escape to host bash, escapes also can be used (eg. \&\&)
	run: 'apt-get update && \
			apt-get install -y curl'

	# run a command inside container and redirect output to host
	run: ls -la > myfile

	# run a command inside container and redirect output to container
	run: 'ls -la > myfile'

Includes

Is possible create jobs or fragments of jobs and reuse it in another projects, like a code package library.

There are two major behavior of includes:

Explicit

A fragment of job (eg. define) and needs to be called explicitly to work.

TAG=latest

define docker_build =
	@$(dkr)
	dind: docker:19
	run: docker build -t $(REGISTRY)/$(PROJECT)/$(REPOS):$(TAG) .
endef

All definitions will be load at start of makefile, after it is possible to call at your custom job.

my-custom-build:
	$(docker-build)

Implicit

Just a full job in another project.

TAG=latest

docker_build:
	@$(dkr)
	dind: docker:19
	run: docker build -t $(REGISTRY)/$(PROJECT)/$(REPOS):$(TAG) .

The jobs will be load at start and can be called directly.

make docker_build

• No needs to implement the job at main Makefile.
• Very useful for similar projects.

mkdkr.csv

A file with name mkdkr.csv, that contains the list of remote includes.

Needs to be at same place o main Makefile.

commitlint,https://github.com/rosineygp/mkdkr_commitlint.git,master,main.mk
docker,https://github.com/rosineygp/mkdkr_docker.git

The file contains four values per line in following order

#	Name	Definition
1	alias *	unique identifier of include and clone folder destiny
2	reference *	any git clone reference
3	checkout	branch, tag or hash that git can checkout (default master)
4	file	the fragment of Makefile that will be included (default main.mk)

* required

Collection

• docker
• commit lint
• exporter

Small collection, use it as example

Examples

Simple

simple:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr!"

Is possible to mix images during job, see in example

Makefile

Service

service:
	@$(dkr)
	service: nginx
	instance: alpine
	run: apk add curl
	run: curl -s nginx

Makefile

DIND

Privileged job

dind:
	@$(dkr)
	dind: docker:19
	run: docker build -t project/repos .

Makefile

Escapes

pipes:
	@$(dkr)
	instance: ubuntu:18.04
	run: "find . -iname '*.mk' -type f -exec cat {} \; | grep -c escapes"

More examples at file

Makefile

Shell

Switch to another shell

shell:
	@$(dkr)
	instance: ubuntu
	export MKDKR_SHELL=bash
	run: 'echo $$0'

More examples at file

Makefile

Stdout

Get last command output

Use to filter or apply some logic in last command executed (also outside container)

stdout:
	@$(dkr)
	instance: alpine
	run: echo "hello mkdkr!"
	cat "$(MKDKR_JOB_STDOUT)"

$(MKDKR_JOB_STDOUT) return path of file

stdout:
	@$(dkr)
	instance: debian
	run: apt-get update
	run: apt-get install curl -y
	run: dpkg -l
	$(stdout) | grep -i curl && echo "INSTALLED"

$(stdout) return output file using cat

Makefile

Pipeline

Group of jobs for parallel and organization execution

pipeline:
	make test -j 3	# parallel execution
	make build
	make pack
	make deploy

Makefile

Environment Variables

Name	Default	Description
MKDKR_TTL	3600	The time limit to a job or service run
MKDKR_SHELL	sh	Change to another shell eg. bash, csh
MKDKR_JOB_STDOUT	last stdout	Path of file, generated with last stdout output
MKDKR_JOB_NAME*	(job|service)_target-name_(uuid)	Unique job name, used as container name suffix
MKDKR_INCLUDE_CLONE_DEPTH	1	In the most of case you no need change history for includes
MKDKR_BRANCH_NAME		Return current git branch, if it exist
MKDKR_BRANCH_NAME_SLUG		Return current git branch, if it exist, with safe values

• to overwrite the values use: export <var>=<value>
• * auto generated

Migration

Migration from release-0.26, just execute the following script on your terminal at root of your project.

curl https://raw.githubusercontent.com/rosineygp/mkdkr/master/.mkdkr > .mkdkr

mkdkr_migration() {
  sed -i 's/\.\.\.\ job/instance:\ /g;s/\.\.\.\ service/service:\ /g;s/\.\.\.\ privileged/dind:\ /g;s/\.\.\.\ /instance:\ /g;s/\.\.\ /run:\ /g;s/@\$(\.)/@\$(dkr)/g' ${1}
}

export -f mkdkr_migration

mkdkr_migration Makefile

find . -iname *.mk -exec bash -c 'mkdkr_migration "$0"' {} \;