S3 event improvements

[cmacduff]

Summary

The S3 events functionality should let me add filters in the subscription in the SNS topic. Ideally I could set multiple lambda's to listen to the same bucket that listen to different prefixes.

Motivation

The setup of "S3 Event -> SNS" is great, however there isn't a way to set filters within the subscription in the SNS event. Ideally I could send different events to different lambda's based on the prefix of the S3 object.

Take for example a bucket of logs that has both Cloudfront and S3 access logs. The cloudfront access logs go to s3://log_bucket/cloudfront_logs and S3 access logs go to s3://log_bucket/s3_logs. There should be one notification from s3://log_bucket to the SNS topic, that has two subscription filtering on the 2 different prefixes to trigger events in 2 different lambdas.

Guide-level explanation

This would be a more fine tuned way to control this traffic compared to the application wide "S3 Event Configuration" settings. if I look at how the SNS configurations work, I think something like this would work nicely:

s3_event "my-bucket", filter_policy: { prefix: "prefix01" }
def handle_prefix01_events
  ...
end

Instead of setting the filters on the S3 notification that the "S3 Event Configuration" does from config/application.rb, this approach would be setting the filter in the SNS subscription to the topic that is getting created.

Reference-level explanation

This might need to loop into the shared resource setup as well. I could see requiring the usage of a shared sns topic to achieve this, especially if you did multiple lambda's. In such a case, maybe this approach would make more sense:

s3_event sns: ref(:shared_sns_topic_name), filter_policy: { prefix: "prefix01" }
def handle_prefix01_events
  ...
end

This would make it easier to distinguish between the current global config approach on the S3 event notification and the mapping to the individual SNS topic.

Drawbacks

more complexity in the S3 feature

Unresolved Questions

not sure.

misc other

I think I could achieve 95% of this with the framework as is. Instead of using the s3_event for my lamba's, I'd use sns_topic as a shared resource and adding the filtering with that. The only piece I'd have to set myself is the s3 event -> shared sns topic. Wouldn't be that bad, in fact that approach might make it harder to justify the need to do this. Maybe Jets just needs a way to configure notification to an SNS topic with the S3 event being an example of one?

[SteffanPerry]

This is probably a good feature to add, we use this a lot via custom shared resources. ill try to knock this out soon ish. I believe you can already kind of do this our of the box, but we may need to update the iam policies.

UPDATE: S3 requires only a small change, this would also be very beneficial in sqs_events (where a SQS queue is subscribed to SNS topic). SNS events seems to already be able to accept this configuration.