403 Forbidden Exception when using this from an ec2 instance

[ericycfu]

Hi,

I have an ec2 instance with the following details

AMI: Amazon Linux 2
Python: 3.10.5
requests-ip-rotator: 1.0.12

My local mac computer has the same configuration with MacOS Monterey 12.5

I follow the code from the documentation.

gateway = ApiGateway("some_website.com", regions=["us-east-2"])
gateway.start()
session = requests.Session()
session.mount("some_website.com", gateway)

# Several times with slightly different payloads
session.request(method, url, headers=headers, data=payload) 

The code works locally, and also when connected to an AWS VPN. However, when run it from my EC2 instance, I get the following error (excerpt from my logs)

status_code=403, 
response_text={"message":"Forbidden"}, 
response_headers = {'Date': 'Wed, 17 Aug 2022 01:41:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '23', 'Connection': 'keep-alive', 'x-amzn-RequestId': '50fe02af-18a2-42ca-adc0-ed76c3432279', 'x-amzn-ErrorType': 'ForbiddenException', 'x-amz-apigw-id': 'W_AP1ZWIiYdDlQw='}

I'm using the same credentials locally and on ec2. For my ec2 instance's IAM policy, I've also added the permissions

  {
    "Action" : "apigateway:*",
    "Effect" : "Allow",
    "Resource" : "*"
  }

As part of my experimenting, I also updated the resource policy for the generated API gateway to the following

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "execute-api:*",
            "Resource": "*"
        }
    ]
}

The security group for my ec2 instance also allows all outbound traffic.

  egress {
    from_port        = 0
    to_port          = 0
    protocol         = "-1"
    cidr_blocks      = ["0.0.0.0/0"]
    ipv6_cidr_blocks = ["::/0"]
  }

Do you have any idea why this might be happening?

[ericycfu]

Tested out on google.com and it worked. Concluded that the site I was trying to scrape had implemented an AWS WAF that blocked these requests.