CloudLog

Logs agregation system built on top of AWS with Python, TypeScript and React

System Architecture

System contains from for base elements:

Logger deamon to install on PC
Log handler for existing python applications
LogSniffer client React application for logs exploration
Backend built on AWS with Terraform

Backend is based on serverless AWS resources. Thanks to use of Terraform, we can easily create, modify and delete infrastrucutre. The definition You can find in infra folder. Backend consists of 6 main elements excluding IAM roles:

REST API Gateway.
S3 bucket used as a hosting for the LogSniffer app.
DynamoDB database used to store the logs.
SQS Queue logs_buffer used as a buffer for incomming requests.
Lambda function save_logs that accepts incomming requests from the SQS queue and save the logs in the database.
Lambda function read_logs used to read the logs

API

Deployment

Deploying the application is very easy. All You have to do to get AWS credentials for terraform and place it in /home/{USER}/.aws/credentials file, and execute following commands on the deploy.sh script:

chmod +x ./deploy.sh #adding permissions
sh ./deploy.sh #execute the script

Deployment script will perform the following actions:

It will check if all required binaries are installed on Your machine.
Run a tests.
Plan and deploy the infrastructure.
Deploy all services.

Logs mapping

Log format

os: string - [Windows,Linux]
severity: number <0-7>
message: string
timestamp: number
hostname: string - workstation name
unit: string - process/application name
type: string - [system, application, logger]
raw: string - raw format of log

OS logs mapping

field	Windows	Linux
os	"windows"	"linux"
severity	Level	PRIORITY
message	Message	MESSAGE
timestamp	TimeCreated	__REALTIME_TIMESTAMP
hostname	MachineName	_HOSTNAME
unit	ProviderName	_EXE
type	ContainerName	`application` if ("opt" or "snap") in _EXE else `system`
raw	*	*