SIEM on Amazon Elasticsearch Service (Amazon ES) is a solution that collects multiple types of logs from AWS multiple accounts, correlates and visualizes the logs to investigate security incidents. Deployment is performed with AWS CloudFormation or AWS Cloud Development Kit (AWS CDK). Deployment will finish within about 20 minutes. When AWS services put the log to Simple Storage Service (Amazon S3), AWS Lambda function will load log to Amazon ES. Then you can visualize logs with dashboard and correlate the multiple logs to investigate security incident.
SIEM on Amazon ES can load and correlate following logs.
AWS Service | Log |
---|---|
AWS CloudTrail | CloudTrail Log Event |
Amazon Virtual Private Cloud (Amazon VPC) | VPC Flow Logs |
Amazon GuardDuty | GuardDuty findings |
AWS Security Hub | Security Hub findings GuardDuty findings Amazon Macie findings Amazon Inspector findings AWS IAM Access Analyzer findings |
AWS WAF | AWS WAF Web ACL traffic information AWS WAF Classic Web ACL traffic information |
Elastic Load Balancing | Application Load Balancer access logs Network Load Balancer access logs Classic Load Balancer access logs |
Amazon CloudFront | Standard access log Real-time log |
Amazon Simple Storage Service (Amazon S3) | access log |
Amazon Route 53 Resolver | VPC DNS query log |
Linux OS via CloudWatch Logs |
/var/log/messages /var/log/secure |
Amazon Elastic Container Service (Amazon ECS) via FireLens |
Framework only |
Supported logs are normalized according to the Elastic Common Schema. Please refer to here to see correspondence log table of original and normalized field name.
See here
Deploy SIEM on Amazon ES domain for public access using a CloudFormation template. Please refer to Advanced Deployment for deploying within the Amazon VPC and customizing.
You can add country information and latitude and longitude location information to the IP address. For location information, SIEM on Amazon ES will download GeoLite2 Free from MaxMind and will use it. If you would like to give location information, please obtain a free license from MaxMind.
Note: CloudFormation deploys Amazon ES with t3.small.elasticsearch instance. Please change the instance type when using SIEM in production. In addition, please use AWS Management Console to add or change instance type, change Amazon ES node configuration. Using CloudFormation template is installed only, it does not manage changes, deletions, etc. of Amazon ES domain.
You can deploy with following CloudFormation template.
Region | CloudFormation |
---|---|
N. Virginia (us-east-1) | ![]() |
Oregon (us-west-2) | ![]() |
Tokyo (ap-northeast-1) | ![]() |
Frankfurt (eu-central-1) | ![]() |
London(eu-west-2) | ![]() |
If your region doesn't list above, use manually this template.
https://aes-siem-<REGION>.s3.amazonaws.com/siem-on-amazon-elasticsearch.template
Or you can create your template with following procedure.
You can skip this session if you already deploy with above CloudFormation template.
The following procedures assumes that all of the OS-level configuration has been completed. They are:
- Amazon EC2 instance running Amazon Linux 2
- "Development Tools"
- Python 3.8
- Python 3.8 libraries and header files
- git
sudo yum groupinstall -y "Development Tools"
sudo yum install -y amazon-linux-extras
sudo amazon-linux-extras enable python3.8
sudo yum install -y python38 python38-devel git jq
sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1
sudo update-alternatives --install /usr/bin/pip3 pip3 /usr/bin/pip3.8 1
Clone the SIEM on Amazon ES from GitHub repository:
git clone https://github.com/aws-samples/siem-on-amazon-elasticsearch.git
export TEMPLATE_OUTPUT_BUCKET=<YOUR_TEMPLATE_OUTPUT_BUCKET> # Name for the S3 bucket where the template will be located
export AWS_REGION=<AWS_REGION> # region where the distributable is deployed
Note: You must manually create a bucket in S3 called $TEMPLATE_OUTPUT_BUCKET to copy the distribution. The assets in bucket should be publicly accessible. The build-s3-dist.sh script DOES NOT do this
cd siem-on-amazon-elasticsearch/deployment/cdk-solution-helper/
chmod +x ./step1-build-lambda-pkg.sh && ./step1-build-lambda-pkg.sh && cd ..
chmod +x ./build-s3-dist.sh && ./build-s3-dist.sh $TEMPLATE_OUTPUT_BUCKET
aws s3 cp ./global-s3-assets s3://$TEMPLATE_OUTPUT_BUCKET/ --recursive --acl bucket-owner-full-control
aws s3 cp ./regional-s3-assets s3://$TEMPLATE_OUTPUT_BUCKET/ --recursive --acl bucket-owner-full-control
Deploy with https://s3.amazonaws.com/$TEMPLATE_OUTPUT_BUCKET/siem-on-amazon-elasticsearch.template
It will probably take about 20 mins to finish deploy Amazon ES. Then you will configure Kibana.
- To login Amazon ES, move to CloudFormation console, select the stack and "Outputs" in tab menu. Then you can see Kibana's username, password and URL.
- To import Kibana's configuration such as dashboard, download saved_objects.zip. Unzip the file.
- Go to Kibana console. Click "Management" in left side menu, "Saved Objects", "Import" and "Import". Select unzip file dashboard.ndjson. Then logout once to load the configuration.
PUT logs to S3 Bucket, aes-siem-<YOUR_AWS_ACCOUNT>-log. Then the log will be loaded to Amazon ES. You can see more details instructions for each logs.
Please use AWS Management Console to change the access policy of Amazon ES, storage size, node instance, Availability Zone, UltraWarm, etc.
Amazon ES indexes each log and rotates it once a month. You can change this term.
You can load non AWS resources to Amazon ES in this framework. See Configure SIEM fore more details
The es-loader, python script, can load stored old logs in S3 Bucket to Amazon ES. Just execute es-loader on Local environment.
AWS Resource | Resource Name |
---|---|
Amazon ES 7.X | aes-siem |
S3 bucket | aes-siem-[AWS_Account]-log |
S3 bucket | aes-siem-[AWS_Account]-snapshot |
S3 bucket | aes-siem-[AWS_Account]-geo |
Lambda function | aes-siem-es-loader |
Lambda function | aes-siem-deploy-aes |
Lambda function | aes-siem-configure-aes |
Lambda function | aes-siem-geoip-downloader |
Lambda function | aes-siem-BucketNotificationsHandler |
AWS Key Management Service (AWS KMS) CMK & Alias |
aes-siem-key |
Amazon SQS Queue | aes-siem-dlq |
Amazon SQS Queue | aes-siem-sqs-splitted-logs |
CloudWatch Events | aes-siem-CwlRuleLambdaGeoipDownloader |
Amazon SNS Topic | aes-siem-alert |
Amazon SNS Subscription | entered email |
- Delete aes-siem stack in CloudFormation
- Delete following AWS resources manually.
- aes-siem domain (Amazon ES)
- aes-siem-123456789012-log (Amazon S3)
- aes-siem-123456789012-geo (Amazon S3)
- aes-siem-123456789012-snapshot (Amazon S3)
- aes-siem-key of CMK alias and key (AWS Key Management Service)
- BE CAREFULL. You will not be able to read the logs, if you encrypt logs with this key.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.
This product uses GeoLite2 data created by MaxMind and licensed under CC BY-SA 4.0, available from https://www.maxmind.com.