/serverless-cloudwatch-logs-exporter

AWS Serverless Lambda function that sends log data from CloudWatch Logs and S3 🎓

Primary LanguagePython

Serverless Cloudwatch Logs To S3 Exporter

AWS Serverless Lambda function that sends log data from CloudWatch Logs and S3.

Fig : Serverless Cloudwatch Logs To S3 Exporter

Follow this article in Youtube

  1. Pre-Requisities

    We will need the following pre-requisites to successfully complete this activity,

    • S3 Bucket - Update the bucket with the policy mentioned below
    • IAM Role serverless-cw-to-s3-exporter-role - Get Help for setting up IAM Role
      • Managed Permissions: AWSLambdaExecute - To allow Lambda to log events & Write to S3 Buckets
      • Inline Policy: ListBucketPolicy mentioned below
    • CloudWatch Log Group Name[s]

  2. S3 Bucket Policy

    To receive CloudWatch Logs exports the following bucket policy should be applied.

    {
    "Version": "2008-10-17",
    "Id": "Policy1335892530063",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.YOUR-REGION.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "logs.YOUR-REGION.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::YOUR-BUCKET-NAME/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        }
    ]
}

  3. Lambda IAM Policy

    Create a new IAM Policy ListBucketPolicy with the following configuration. If you want, restrict it only to the buckets of your interests, in resources section. A policy to allow the script to verify valid S3 buckets

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

  4. ⚙️ Configure Lambda Function - Serverless CloudWatch Logs To S3 Exporter

    • Provide the function name & choose Python 3.7
    • Attach the IAM Role - serverless-cw-to-s3-exporter-role
    • Get code here
    • Save the lambda function

  5. ⚙️ Configure Lambda Triggers

    Goto the Cloudwatch Dashboard, We are going to use Event Rules

    • Choose Create a new Rule
    • Choose Schedule - Get Help
    • For Target, Choose Lambda Function
      • From dropdown select your Lambda Function Name
    • In the bottom, Configure Details
    • Fill the Rule Name & Rule Description
      • Make sure it is Enabled
    • Enable Trigger by Checking the box
    • Click Save

    Now your lambda function should be triggered based on your schedule

  6. 🔬 Testing the solution

    You can test it manually or set the cloudwatch events to trigger closest to the current UTC time

Lamdba Runtime

Increase your lambda runtime as necessary depending about the amount of data to move. Perhaps run it inside step functions.

Programming language

Python 3.x

Known Limitations

This Lambda function has the following limitation:

  • Currently If you want to export all log groups, you have to list them all manually. This is by design, as the export tasks are limited to 1 PENDING task at any time by AWS, and Lambda have limited time to push all the logs.

👋 Buy me a coffee

Buy me a coffee ☕ through Paypal, or You can reach out to get more details through here.

📚 References

[1] - AWS Docs - Export Log Data to Amazon S3

[2] - CloudWatch - Schedule Expressions for Rules

ℹ️ Metadata

Level: 200