/aws-alb-logs-to-elasticsearch

Primary LanguageJavaScript

Send ALB logs from S3 bucket to ElasticSearch using AWS Lambda.

This project based on awslabs/amazon-elasticsearch-lambda-samples Sample code for AWS Lambda to get AWS ALB log files from S3, parse and add them to an Amazon Elasticsearch Service domain.

Deployment Package Creation

  1. On your development machine, download and install Node.js.

  2. Go to root folder of the repository and install node dependencies by running:

    npm install

    Verify that these are installed within the node_modules subdirectory.

  3. Create a zip file to package the index.js and the node_modules directory

The zip file thus created is the Lambda Deployment Package.

AWS Configuration

Set up the Lambda function and the S3 bucket. You can reffer to for more details > Lambda-S3 Walkthrough.

Please keep in mind the following notes and configuration overrides:

  • The S3 bucket must be created in the same region as Lambda is, so that it can push events to Lambda.

  • When registering the S3 bucket as the data-source in Lambda, add a filter for files having .log.gz suffix, so that Lambda picks up only alb log files.

  • The following authorizations are required:

    1. Lambda permits S3 to push event notification to it
    2. S3 permits Lambda to fetch the created objects from a given bucket
    3. ES permits Lambda to add documents to the given domain
    4. Lambda handler is set to index.handler
    5. Lambda environment variable es_endpoint to the elasticseach endpoint (dom't include https://)

    The Lambda console provides a simple way to create an IAM role with policies for (1).
    For (2), when creating the IAM role, choose the "S3 execution role" option; this will load the role with permissions to read from the S3 bucket.
    For (3), add the following access policy to permit ES operations to the role.

{
      "Sid": "AllowLambdaAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/lambda_s3_exec_role"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:eu-west-1:123456789012:domain/elastic-search-domain/*"
}

Terraform Module

for terraform module to deploy this lambda see neillturner/alb-logs-to-elasticsearch

module "vpc_alb_logs_to_elasticsearch" {
  source        = "neillturner/alb-logs-to-elasticsearch/aws"
  version       = "0.1.0"

  prefix        = "vpc_es_"
  es_endpoint   = "vpc-gc-demo-vpc-gloo5rzcdhyiykwdlots2hdjla.eu-central-1.es.amazonaws.com"
  s3_bucket_arn = "arn:aws:s3:::XXXXXXX-elb-logs-eu-west-1"
  s3_bucket_id  = "XXXXXXX-elb-logs-eu-west-1"
  subnet_ids    = ["subnet-d9990999"]
}

Event source Add Event source for your lambda function

Event source type: S3

Bucket: s3-alb-access-logs

Event type: Object Created (All)

Prefix: public-alb/AWSLogs/123456789012/elasticloadbalancing/eu-west-1/

Suffix: .log.gz

#License ASL

https://aws.amazon.com/asl/