s12v/awsbeats

CloudWatch Logs output plugin

mumoshu opened this issue · 2 comments

mumoshu commented

As per me and @s12v's discussion in #8, probably we can include CloudWatch Logs output plugin within awsbeats.
Anyone has a specific use-case for that? Any concrete feature you'd like to have?

jpuskar commented

I'd love for filebeat to output to cloudwatch logs! The cloudwatch logs agent doesn't have a 'processors' feature, or really any concept of fields. This makes adding cloud metadata to events really difficult.

My current experiment is to try filebeat -> kinesis -> lambda -> cwlogs -> kinesis -> logstash -> es.
A CWLogs output plugin would skip the first kinesis -> lambda.

mumoshu commented

@jpuskar Hey, thanks for the feedback!

I'm also looking forward to this feature, especially after the arrival of CloudWatch Logs Insight which gives us the serverless alternative to kibana/es query :)

Your experiment looks good. The pipeline from kinesis to lambda should be used whenever you can't afford the cloudwatch logs' cost per ingested GB - $0.4~0.5/GB oh my - and the ingestion rate linit.

But it doesn't negate the usefulness of this feature for the smaller scale.