DEPRECATED
This is no longer supported, it is moved to wix/supraworker
Supraworker - Pull Jobs from Anywhere
The abstraction layer around jobs, allows pull a job from any API, call-back your API, observe execution time and to control concurrent execution.
It's responsible for pulling the commands(jobs) from your API, running commands, and streaming the logs back to your API. It also sends state updates to remote API.
Getting started
Prerequisite:
- API service that serve jobs:
- You can check the simple implantation of the API service written ib Python
- Supraworker configuration:
- You can check the configuration for the above API service written ib Python
- Running supraworker on a server
Installation
MacOs X
Binary installation
Homebrew is a free and open-source package management system for Mac OS X.
brew tap weldpua2008/tap
brew install weldpua2008/tap/supraworker
To update to the latest, run
brew upgrade weldpua2008/tap/supraworker
Installation from source code
- Find the version you wish to install on the GitHub Releases
page and download either the
darwin-amd64
binary for MacOS or thelinux-amd64
binary for Linux. No other operating systems or architectures have pre-built binaries at this time.
NOTE: Running releasses on MacOs: You need to download file, extract it and remove attributes with the following command (where ~/Downloads/supraworker_darwin_amd64/supraworker is Path to the file)
$ xattr -d com.apple.quarantine ~/Downloads/supraworker_darwin_amd64/supraworker
$ ~/Downloads/supraworker_darwin_amd64/supraworker
Linux
Download the latest release
curl --silent -L "https://api.github.com/repos/weldpua2008/supraworker/releases/latest" \
| jq --arg PLATFORM_ARCH "$(echo `uname -s`_amd| tr '[:upper:]' '[:lower:]')" -r '.assets[] | select(.name | contains($PLATFORM_ARCH)).browser_download_url' \
| xargs -I % curl -sSL % \
| sudo tar --strip-components=1 -xzf -
Installing from source
- install Go
v1.13+
- clone this down into your
$GOPATH
mkdir -p $GOPATH/src/github.com/weldpua2008
git clone https://github.com/weldpua2008/supraworker $GOPATH/src/github.com/weldpua2008/supraworker
cd $GOPATH/src/github.com/weldpua2008/supraworker
- install golangci-lint for linting + static analysis
- Lint:
docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.24.0 golangci-lint run -v
Configuration
Define config at $HOME/supraworker.yaml
:
NOTE: Keys are not case sensitivity (spf13/viper#635 (comment)): Viper's default behaviour of lowercasing the keys for key insensitivity is incompatible with these standards and has the side effect of making it difficult for use cases such as case sensitive API credentials in configuration. For eg: MyApiKey=MySecret
# ClientId in case you need to identify the worker
clientId: "my_uniq_client_id"
# how ofen call your API
api_delay_sec: 10
# jobs related operations
jobs:
get:
url: "http://localhost:80/get/new/job"
method: POST
headers:
"Content-type": "application/json"
params:
"clientid": "{{ .ClientId}}"
Use Cases
- Airflow sends task for an execution on AWS EMR
- Building your CI/CD system
Running tests
- expires all test results
$ go clean -testcache
- run all tests
$ go test -bench= -test.v ./...