SQS not deleteing
goochi1 opened this issue · 1 comments
We pulled your latest changes and one of the developers is haveing an issue with the Que:
Background
We have encountered some issue with the latest update of the Binary Alert analyzer function.
The behavior of this issue is we are getting duplicated scanning notification every 5min.
To reproduce the issue, you can deploy the v0.6 Binary Alert terraform module. Subscribe to the SNS topic with test email, and upload a file to the S3 bucket, this will cause a new message added in SQS queue.
After file being analyzed, an SNS notification is published to the topic. And after every 5min, the SQS triggers the lambda function again even when there are no new files uploaded to s3, and I got the notification from SNS with the same file.
Seems the issue is the analyzer lambda function could not delete the analyzed SQS message.
This is what we have in the lambda log:
[INFO] 2018-07-12T13:57:14.523Z 759fa9ea-85db-11e8-8dd9-2b2990818c65 Analyzing "test-k-analyser-bucket:google day.docx"
[INFO] 2018-07-12T13:57:14.527Z 759fa9ea-85db-11e8-8dd9-2b2990818c65 Resetting dropped connection: test-k-analyser-bucket.s3.eu-west-1.amazonaws.com
[INFO] 2018-07-12T13:57:15.58Z 759fa9ea-85db-11e8-8dd9-2b2990818c65 S3:test-k-analyser-bucket:google day.docx did not match any YARA rules
[INFO] 2018-07-12T13:57:15.59Z 759fa9ea-85db-11e8-8dd9-2b2990818c65 Publishing an SNS alert
[INFO] 2018-07-12T13:57:15.87Z 759fa9ea-85db-11e8-8dd9-2b2990818c65 Starting new HTTPS connection (1): monitoring.eu-west-1.amazonaws.com
END RequestId: 759fa9ea-85db-11e8-8dd9-2b2990818c65
REPORT RequestId: 759fa9ea-85db-11e8-8dd9-2b2990818c65 Duration: 9232.94 ms Billed Duration: 9300 ms Memory Size: 1024 MB Max Memory Used: 316 MB
This might happen because of the “receipts_to_delete” condition added with the latest change from the main.py file:
# Delete all of the SQS receipts (mark them as completed).
receipts_to_delete = [msg['receipt'] for msg in event.get('messages', [])]
if receipts_to_delete:
analyzer_aws_lib.delete_sqs_messages(event['queue_url'], receipts_to_delete)
I have now rolled back to the previous version of the analyzer lambda function and the SQS cleanup works as expected now.
Can you please help look into this issue? Thanks very much.
Lambda can now automatically delete messages off of an SQS queue once they have been processed! This should fix any problems with the queue, although keep in mind every message in a batch has to be processed successfully in order for it to be deleted