/background-jobs

A demo app to simulate background job processing

Primary LanguageGoApache License 2.0Apache-2.0

background-jobs

A demo app to simulate background job processing

Getting started

This solutions is composed out of three components:

  1. Background Jobs: A web server with an API for Creating and Getting Jobs.
  2. Backgournd Processing: A infinite process that processes the background jobs introduced by the previous componet. See source.
  3. Background commons: it is a dependency to share logic between above components. See source.

The two services are cloud-native: containerized and ready to deploy into a kubernetes cluster.

How to install and deploy the components?

Both components follow the same structure.

  1. Dockerfile to containerize the application.
  2. create-docker-image.sh which make the container image out of the source code.
  3. build.yaml Which contains all the artifacts to run the component on kubernetes.

A typical installation is:

chmod +x create_docker-image.sh
./create_docker-image.sh
kubectl apply -f build.yaml

Installing the database

This solution requires a database to persist the different states of a job. In order to install it, the Background commons repo contains a helm chart with a MariaDB installation. The DB is installed into mariadb namespace.

kubectl apply -f mariadb-helm-chart.yaml

Note: In the case of the web server (bg-jobs) there is a service to expose the API in and out of the cluster.

How to expose the web server?

The service is a NodePort kind. Therefore, in order to send traffic from outside of the cluster you need two things: cluster IP and node port.

export CLUSTER_IP=$(minikube ip)
export PORT=$(kubectl get svc background-jobs -n background-jobs -o go-template='{{(index .spec.ports 0).nodePort}}')
echo $CLUSTER_IP:$PORT

Note: This has only been tested using a minikube cluster.

API

There are two endpoints available:

  1. GET /jobs/id: that returns a json with all the available information of a job.
  2. POST /objects/{object_id:[0-9]+}/jobs/create: that creates a job with an object associated. Only 1 job every 5min for the same object_id.

Test examples:

# Creating a job
curl -X POST http://$CLUSTER_IP:$PORT/objects/14/jobs/create
# Checking its status
curl http://$CLUSTER_IP:$PORT/jobs/362

TODO:

  • Implement retries.
  • Fix unit testing. It uses shared test db.
  • Add e2e tests.
  • Add a UI to see the status of the jobs.
  • Use a migration tool to manage the evolution of the database.
  • Fix timezone alignment in development.
  • Add configmap to deal with settings such as timeouts, db, thresholds...