µ-batching

Overview

A simple microbatching system in Go, designed to process jobs efficiently in batches. It includes a queue for job management and a batch processing mechanism. The system also provides a REST API to submit jobs, check job status, and retrieve information about queued and processed jobs.

Features

Job Submission: Submit jobs for processing via a REST API.
Job Queueing: Jobs are queued before being batch processed.
Batch Processing: Jobs are processed in configurable batch sizes and intervals.
Job Status Tracking: Track the status of submitted jobs, whether they're queued or processed.
Job ID Retrieval: Retrieve the IDs of all jobs currently in the queue.
Processed Job Retrieval: Retrieve all processed jobs.

Getting Started

Prerequisites

Go (version 1.15 or later)

Installation & hey, just run it

Clone the repository to your local machine:

git clone https://github.com/1x-eng/mbat.git
cd mbat
touch ./.env # edit the config as you see fit

# build and run the executable
go build -o ./bin/mbat ./cmd/
./bin/mbat

API Endpoints

Submit Job: POST /job/submit
Job Status: GET /job/status/:jobID
Queued Jobs: GET /jobs/queue
Processed Jobs: GET /jobs/processed

Swagger / OpenAPI

µ-batching uses swaggo cli to generate swagger specs. Install swaggo cli using - go install github.com/swaggo/swag/cmd/swag@latest
To generate or refresh swagger specs, swag init -g ./pkg/server/handlers.go
µ-batching also uses fiber-swagger to render swaghger UI.
Once the project is built and run, visit <api>/swagger to view swagger UI.

TODO

Integrate github actions / promote CI/CD and abstract swaggo / fiber-swagger
A whole lot of tests :P