DataReply/alertmanager-sns-forwarder

Constant 503 errors

Closed this issue · 2 comments

Steps you follow to reproduce the error:

Create an IAM role for the sender including sns:Publish. The sender is an ec2 instance running alertmanager-sns-forwarder in a container. Start the container with the appropriate ARN prefix for a topic known to be working. Watch the logs and see 503 errors over and over again (I changed only the customer identifier in the output below):

[GIN] 2019/06/12 - 23:34:51 | 503 |    1.865223ms |      172.17.0.1 | POST     /alert/arn:aws:sns:us-east-2:9999999999:us-east-2-dev-notify
WARN[0099] NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[GIN] 2019/06/12 - 23:35:28 | 503 |    1.802154ms |      172.17.0.1 | POST     /alert/arn:aws:sns:us-east-2:9999999999:us-east-2-dev-notify
WARN[0100] NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[GIN] 2019/06/12 - 23:35:29 | 503 |    1.653303ms |      172.17.0.1 | POST     /alert/arn:aws:sns:us-east-2:9999999999:us-east-2-dev-notify
WARN[0100] NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

What did you expect to happen instead:
I can see the proper formed messages in debug mode submitting to the correct arn. I'd love to see the topic actually publish something.

What's your theory on why it isn't working:
I really don't know. I don't see any failed messages in cloudwatch logging the topic, so I have to believe the full URL which must be needed to publish is somehow wrong inside the container.

It does seem that the messages seem to duplicate the arn prefix.

DEBU[0401] Using topic ARN: arn:aws:sns:us-east-2:9999999999:arn:aws:sns:us-east-2:9999999999:us-east-2-dev-notify

This was caused by the upper limit of the number of SNS alarms per unit time being reached.