This package has been created as a way to deploy a containerized central logging server quickly. The hierarchy is meant as a modular system. Logs are maintained on the central and accessible outside of the docker container.
See : https://github.com/guardsight/gsvsoc_docker-system-of-record/wiki/Roadmap
This is a snapshot to help understand what is going on. Assume overview is running as a docker service on a physical host or virtual machine.
------------------------------------------ | | 601/TCP | ------------- | 514/UDP | | Syslog-ng | | Syslog In ---> | ---> | Docker | | | | Container | | | ------------- | | | | Flat File | <--------------| | /logs/HOSTS/$IP/$Date/* | | | | | ------------- | | --->| Optional | | Optional Access Calls | | Docker | | <------- | |Containers | | | ------------- | | Elastic | | Kibana | | Fleet | ------------------------------------------
- Memory : At minimum 8GB of ram. Provision additional RAM as needed.
- Processor : At minimum 4 cores.
- Sudo access to a debian linux box
- Internet access to download software
- The instructions below are for an Ubuntu 16+ distribution, they can be modified for a CentOS installation.
- Run Installation
sudo bash ./setup-system-of-record.sh
-
Install latest updates and docker
sudo apt update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt update
sudo apt-get install -y docker-ce
-
Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.26.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
-
Prep directories
sudo mkdir -p /opt/secops
sudo mkdir -p /logs/HOSTS
-
Install git and jq
sudo apt install -y git jq
- Make sure you have enough vm mappings available
sudo sysctl -w vm.max_map_count=262144
- Create a local logs group
sudo groupadd -g 1111 logs
- Create a local user and add to logs group
sudo useradd -u 1111 -g 1111 logs
- Change ownership and permisisons of /logs and /opt/secops directories
sudo chown -R logs:logs /logs
sudo chmod -R 750 /logs
sudo chown logs:logs /opt/secops
sudo chmod 770 /opt/secops
sudo usermod -aG logs $(whoami)
- Refresh user groups
newgrp logs
- Note: if this is an issue, log off the machine and log back on.
cd /opt/secops
git clone https://github.com/guardsight/gsvsoc_docker-system-of-record.git
cd /opt/secops/gsvsoc_docker-system-of-record
- Add script file to profile.d
sudo cp /opt/secops/gsvsoc_docker-system-of-record/99-infosec.sh /etc/profile.d/
sudo chmod +x /etc/profile.d/99-infosec.sh
cd /opt/secops/gsvsoc_docker-system-of-record
sudo docker-compose -p 'gsvsoc-system-of-record' up --build -d
- The service should be up within 20 seconds
cd /opt/secops/gsvsoc_docker-system-of-record
sudo docker-compose -p 'gsvsoc-system-of-record' down
cd /opt/secops/gsvsoc_docker-system-of-record
sudo bash docker-cleanup.sh
cd /opt/secops/gsvsoc_docker-system-of-record
sudo docker-compose -p 'gsvsoc-system-of-record' down
sudo bash docker-cleanup.sh
git pull
sudo docker-compose -p 'gsvsoc-system-of-record' up --build -d
- The service should be up within 20 seconds
The logs are stored in /logs/HOSTS/{HOST}/{DATE}/messages.log[.gz]
You can zgrep through these logs and correlate or filter down the data you want
NOTE: This data will be there until the data is moved or purged.
To read the messages without using sudoer permissions, you need to be added to the logs group.
1. sudo usermod -aG sudo USERNAME- /etc/cron.daily/GSVSOC-logs-compress.sh
- Runs daily and compresses the messages.log files in /logs/HOSTS
cd /opt/secops/gsvsoc_docker-system-of-record
sudo cp cronjobs/daily/* /etc/cron.daily/
sudo docker logs --follow syslog-ng
- The following is meant to be run from the parent machine
echo "testing_tcp" | nc -w1 127.0.0.1 601
echo "testing_udp" | nc -u -w1 127.0.0.1 514
- Wait at least 20 seconds, and then verify the log has appeared
- Check raw logs for existence
zgrep -sE "testing_((tcp)|(udp))" /logs/HOSTS/*/"$(date +'%F')"/*
- Instruction:W-0021_-_How-To_Create_A_Syslog_Infrastructure