Scraping amazon not working
arpitgoyall opened this issue · 2 comments
arpitgoyall commented
import os
from requests_ip_rotator import ApiGateway
from dotenv import load_dotenv
import requests
load_dotenv()
with ApiGateway(
"https://www.amazon.in",
regions=["eu-west-1", "eu-west-2"],
access_key_id=os.getenv("aws_key_id"),
access_key_secret=os.getenv("aws_key_secret"),
) as g:
session = requests.Session()
session.mount("https://www.amazon.in", g)
response = session.get("https://www.amazon.in/dp/B09PNHN5ZZ")
print(response.status_code)
print(response.request.headers)
print(response.request.url)
This code is returning the following
503
{'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Host': '0h3hjglii9.execute-api.eu-west-1.amazonaws.com', 'X-My-X-Forwarded-For': '43.220.173.104'}
https://0h3hjglii9.execute-api.eu-west-1.amazonaws.com/ProxyStage/dp/B09PNHN5ZZ
I think it is replacing the request URL which is giving the 503 error, what should I do?
MsLolita commented
Did you solved it?
Ge0rg3 commented
Hey @arpitgoyall, thanks for raising the issue. It looks like this is some internal block from Amazon, or some issue with how the requests are coming from their own servers, and not an issue with the library.