Receiving 403 downloading apt signing keys

[StagasaurusRex]

We are receiving a 403 forbidden response when attempting to download the apt signing keys. This is the output from ansible,

TASK [datadog.datadog : Download https://s3.amazonaws.com/public-signing-keys/DATADOG_APT_KEY_CURRENT.public to import key DATADOG_APT_KEY_CURRENT] **********************************
fatal: [atlas-staging-dbi]: FAILED! => {"changed": false, "dest": "/tmp/ansible.ui3l8iudkeys/DATADOG_APT_KEY_CURRENT", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 403: Forbidden", "status_code": 403, "url": "https://s3.amazonaws.com/public-signing-keys/DATADOG_APT_KEY_CURRENT.public"}

If I try to download the keys manually with curl https://s3.amazonaws.com/public-signing-keys/DATADOG_APT_KEY_CURRENT.public I receive what looks a 403 response from S3.

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>M0RC9NWT4E6MA8NA</RequestId><HostId>wvC/52l28hFacJLgoCpUEpRUBM+p3niKbWLppNZ5aOP+9JMOXkLsdsOqlg489Ufp5kPCQUiuzrA=</HostId></Error>

If I download the keys from the keys.datadoghq.com domain, they download fine.

We are only seeing this issue on our EC2 instances located in AWS, all of which are in us-east-1. We are able to download these keys to our local machine. That response looks like an S3 response, which makes me think this is not a networking issue on our end.

[bkabrda]

Hi 👋 I can't reproduce this on a Ubuntu 22.04 machine running in us-east-1. It's remotely possible that there's something wrong with our S3 configuration which prevents cross-account public access, but it would be weird that we wouldn't hear about this earlier, because we've been using this setup for more than a year and nobody has ever complained before. I'll try to bring this up with the folks who maintain our cloud infrastructure to see if they can figure something out. I'll let you know once I have more information.