This request monitoring service logs the time the request was made and other request related details to a MySQL database.
Listener and Worker are seperate code and share data using Redis in-memory database (Redis Lists).
The Listener handles request concurrently and save logs to Redis Lists acting as Queue, and Worker proccess upto 20,000req/min and log them in bulk on MySQL database clearing Redis Queue.
The Worker code can handle Graceful shutdown without loosing any relavent data.
git clone https://github.com/prathamiitp/monitoring_service
Installation and documentation
3. Set GOPATH and GOROOT variables by adding following lines in file: .bashrc for ubuntu inside HOME folder :
export GOPATH=$HOME/go
export GOROOT=/usr/local/go/
export PATH=$PATH:$GOPATH/bin
export PATH=$PATH:$GOROOT/bin
- First Setup MySQL and Redis system servers from database-server-setup-for-system mentioned below.
sudo systemctl start mysql
sudo systemctl start redis-server
To check if the server is up and running, just replace 'start' with 'status'
To stop once work is done, replace 'start' with 'stop'
- MySQL
- Redis (use the APT repository for it includes redis-cli) Remeber to add the database for both Redis and MySQL with the schema and name used in code.
- MySQL: [github.com/go-sql-driver/mysql] , [database/sql]
- Redis: [github.com/go-redis/redis/v9] (Github repo)
- Also check Uptrace's documentation
- optimize cpu utilization (utilization is high because of infinite loop in worker which make connection to redis to check for any new datqa in queue)
- cleaner code implimentation (break logging function from worker code into smaller function, also improve the implimentation of graceful shutdown)
- upload go mod file and create a shell script for all the setup
- update readme with testing method for the monitoring service.
- reduce redis connection in worker, try to take up data in bulk from redis(just like done in case of mysql) which is taken unit by unit for now