How gateway.start() navigate IP?
Raychanan opened this issue · 2 comments
Raychanan commented
Hi! Thanks for your invaluable work. Your package has proven to be immensely helpful to me!
I am conducting an investigation into the manner in which IP addresses are changing within a single session. I had been sending requests subsequent to invoking the gateway.start() function, but to my surprise, I found that there were no reused IP addresses. This was quite unexpected, as I had assumed that each of the ten endpoints that are initialized by gateway.start() would have a fixed IP address when they made a request to showmyip.com.
Would it be possible for you to provide me with a clue as to why, despite making one hundred requests to showmyip.com, none of the ten endpoints show any reuse of IP addresses? This information would be extremely valuable for the implementation of ThreadPoolExecutor() in my forthcoming project.
Thank you very much for your help.
This is my code :
import requests
from requests_ip_rotator import ApiGateway
from concurrent.futures import ThreadPoolExecutor
import time
from bs4 import BeautifulSoup
from collections import Counter
def fetch_search_results(query, gateway, session):
url = f"https://www.showmyip.com/"
response = session.get(url)
print(f"Status code for '{query}': {response.status_code}")
soup = BeautifulSoup(response.text, 'html.parser')
ip_address = soup.find('h2', {'id': 'ipv4'}).text
print(f"IP address for '{query}': {ip_address}")
return ip_address
def main():
# Create the ApiGateway object
gateway = ApiGateway("https://www.showmyip.com/")
gateway.start()
# Assign the gateway to a session
session = requests.Session()
session.mount("https://www.showmyip.com/", gateway)
queries = ["test"] * 100
# Fetch search results using ThreadPoolExecutor
start_time = time.time()
ip_addresses = []
with ThreadPoolExecutor() as executor:
ip_addresses = list(executor.map(lambda q: fetch_search_results(q, gateway, session), queries))
# Print the time taken
print(f"Time taken: {time.time() - start_time:.2f} seconds")
# Count IP address frequencies
ip_count = Counter(ip_addresses)
# Print the IP addresses and their frequencies
for ip, count in ip_count.items():
print(f"{ip}: {count}")
# Shut down the gateway
gateway.shutdown()
if __name__ == "__main__":
main()
This is my output :
Starting API gateways in 10 regions.
Using 10 endpoints with name 'https://www.showmyip.com/ - IP Rotate API' (10 new).
Status code for 'test': 200
IP address for 'test': 3.12.218.211
Status code for 'test': 200
IP address for 'test': 3.101.201.145
Status code for 'test': 200
IP address for 'test': 44.233.55.54
Status code for 'test': 200
IP address for 'test': 13.38.132.218
Status code for 'test': 200
IP address for 'test': 13.38.133.63
Status code for 'test': 200
IP address for 'test': 3.72.168.169
Status code for 'test': 200
IP address for 'test': 3.101.208.155
Status code for 'test': 200
IP address for 'test': 3.98.86.148
Status code for 'test': 200
IP address for 'test': 3.12.218.47
Status code for 'test': 200
IP address for 'test': 13.38.133.188
Status code for 'test': 200
IP address for 'test': 35.93.126.252
Status code for 'test': 200
IP address for 'test': 3.253.175.240
Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 3.12.217.64
Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 3.253.188.151
IP address for 'test': 3.101.176.159
IP address for 'test': 13.51.120.143
Status code for 'test': 200
IP address for 'test': 3.70.195.242
Status code for 'test': 200
IP address for 'test': 44.233.54.93
Status code for 'test': 200
IP address for 'test': 16.171.50.241
Status code for 'test': 200
IP address for 'test': 13.38.133.20
Status code for 'test': 200
IP address for 'test': 3.15.35.106
Status code for 'test': 200
IP address for 'test': 3.99.196.197
Status code for 'test': 200
IP address for 'test': 3.10.17.50
Status code for 'test': 200
IP address for 'test': 13.40.207.83
Status code for 'test': 200
IP address for 'test': 44.233.55.27
Status code for 'test': 200
IP address for 'test': 99.79.126.58
Status code for 'test': 200
IP address for 'test': 13.52.201.50
Status code for 'test': 200
IP address for 'test': 13.48.74.156
Status code for 'test': 200
IP address for 'test': 3.98.86.200
Status code for 'test': 200
IP address for 'test': 3.15.35.190
Status code for 'test': 200
IP address for 'test': 3.70.195.244
Status code for 'test': 200
IP address for 'test': 13.51.120.49
Status code for 'test': 200
IP address for 'test': 13.38.134.122
Status code for 'test': 200
IP address for 'test': 3.253.186.41
Status code for 'test': 200
IP address for 'test': 13.38.133.166
Status code for 'test': 200
IP address for 'test': 3.72.33.167
Status code for 'test': 200
IP address for 'test': 3.123.14.148
Status code for 'test': 200
IP address for 'test': 3.98.87.68
Status code for 'test': 200
IP address for 'test': 34.223.68.66
Status code for 'test': 200
IP address for 'test': 3.216.140.95
Status code for 'test': 200
IP address for 'test': 3.10.17.64
Status code for 'test': 200
IP address for 'test': 3.70.211.100
Status code for 'test': 200
IP address for 'test': 13.48.74.11
Status code for 'test': 200
IP address for 'test': 16.171.51.95
Status code for 'test': 200
IP address for 'test': 3.98.86.128
Status code for 'test': 200
IP address for 'test': 13.40.207.44
Status code for 'test': 200
IP address for 'test': 44.233.54.186
Status code for 'test': 200
IP address for 'test': 3.123.15.98
Status code for 'test': 200
IP address for 'test': 34.223.69.40
Status code for 'test': 200
IP address for 'test': 3.12.219.184
Status code for 'test': 200
IP address for 'test': 3.101.200.125Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 3.101.200.248
IP address for 'test': 3.98.87.23
Status code for 'test': 200
IP address for 'test': 3.145.230.75
Status code for 'test': 200
IP address for 'test': 13.40.205.43
Status code for 'test': 200
IP address for 'test': 3.101.176.31
Status code for 'test': 200
IP address for 'test': 13.38.132.240
Status code for 'test': 200
IP address for 'test': 13.52.201.80
Status code for 'test': 200
IP address for 'test': 3.98.87.137
Status code for 'test': 200
IP address for 'test': 16.171.51.141
Status code for 'test': 200
IP address for 'test': 13.40.205.165
Status code for 'test': 200
IP address for 'test': 44.206.4.84
Status code for 'test': 200
IP address for 'test': 3.101.201.172
Status code for 'test': 200
IP address for 'test': 3.253.186.219
Status code for 'test': 200
IP address for 'test': 3.216.136.243
Status code for 'test': 200
IP address for 'test': 16.171.51.215
Status code for 'test': 200
IP address for 'test': 3.253.187.91
Status code for 'test': 200
IP address for 'test': 13.38.132.174
Status code for 'test': 200
IP address for 'test': 3.238.212.170
Status code for 'test': 200
IP address for 'test': 3.72.168.54
Status code for 'test': 200
IP address for 'test': 3.123.14.159
Status code for 'test': 200
IP address for 'test': 3.145.230.121
Status code for 'test': 200
IP address for 'test': 3.9.94.4
Status code for 'test': 200
IP address for 'test': 3.9.94.209
Status code for 'test': 200
IP address for 'test': 3.253.188.85
Status code for 'test': 200
IP address for 'test': 3.99.196.111
Status code for 'test': 200
IP address for 'test': 13.38.134.181
Status code for 'test': 200
IP address for 'test': 35.90.133.148
Status code for 'test': 200
IP address for 'test': 35.90.132.88
Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 34.223.71.168
IP address for 'test': 3.253.134.86
Status code for 'test': 200
IP address for 'test': 3.70.195.221
Status code for 'test': 200
IP address for 'test': 34.223.70.87
Status code for 'test': 200
IP address for 'test': 13.40.204.249
Status code for 'test': 200
IP address for 'test': 3.253.187.141
Status code for 'test': 200
IP address for 'test': 44.206.5.222
Status code for 'test': 200
IP address for 'test': 44.210.66.35
Status code for 'test': 200
IP address for 'test': 99.79.126.58
Status code for 'test': 200
IP address for 'test': 13.40.204.215
Status code for 'test': 200
IP address for 'test': 3.101.200.3
Status code for 'test': 200
IP address for 'test': 3.72.33.250
Status code for 'test': 200
IP address for 'test': 3.12.217.171
Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 34.223.68.251
IP address for 'test': 3.10.17.66
Status code for 'test': 200
IP address for 'test': 3.15.35.220
Status code for 'test': 200
IP address for 'test': 3.101.176.163
Status code for 'test': 200
Status code for 'test': 200
IP address for 'test': 3.98.87.79
IP address for 'test': 16.171.51.216
Status code for 'test': 200
IP address for 'test': 3.101.200.93
Status code for 'test': 200
IP address for 'test': 3.101.176.234
Time taken: 9.54 seconds
3.72.168.169: 1
13.38.132.218: 1
3.12.218.211: 1
13.38.133.63: 1
44.233.55.54: 1
3.101.201.145: 1
3.101.208.155: 1
3.98.86.148: 1
13.51.120.143: 1
3.253.175.240: 1
3.253.188.151: 1
13.38.133.188: 1
3.12.218.47: 1
35.93.126.252: 1
3.101.176.159: 1
3.12.217.64: 1
3.70.195.242: 1
13.40.207.83: 1
13.38.133.20: 1
16.171.50.241: 1
3.10.17.50: 1
44.233.54.93: 1
44.233.55.27: 1
3.15.35.106: 1
3.99.196.197: 1
13.48.74.156: 1
13.52.201.50: 1
99.79.126.58: 2
13.51.120.49: 1
3.70.195.244: 1
3.98.86.200: 1
3.15.35.190: 1
3.123.14.148: 1
3.72.33.167: 1
13.38.134.122: 1
3.253.186.41: 1
13.38.133.166: 1
34.223.68.66: 1
3.98.87.68: 1
13.48.74.11: 1
3.70.211.100: 1
3.216.140.95: 1
3.10.17.64: 1
16.171.51.95: 1
13.40.207.44: 1
3.123.15.98: 1
44.233.54.186: 1
3.98.86.128: 1
34.223.69.40: 1
3.101.200.248: 1
3.101.200.125: 1
3.12.219.184: 1
13.40.205.43: 1
3.98.87.23: 1
3.145.230.75: 1
13.52.201.80: 1
16.171.51.141: 1
13.38.132.240: 1
3.101.176.31: 1
3.98.87.137: 1
13.40.205.165: 1
3.101.201.172: 1
16.171.51.215: 1
44.206.4.84: 1
3.253.186.219: 1
3.253.187.91: 1
3.216.136.243: 1
13.38.132.174: 1
3.72.168.54: 1
3.238.212.170: 1
3.9.94.4: 1
3.123.14.159: 1
3.9.94.209: 1
3.145.230.121: 1
13.38.134.181: 1
3.253.188.85: 1
3.99.196.111: 1
35.90.133.148: 1
35.90.132.88: 1
34.223.71.168: 1
3.253.134.86: 1
3.70.195.221: 1
13.40.204.249: 1
34.223.70.87: 1
3.253.187.141: 1
44.206.5.222: 1
3.101.200.3: 1
44.210.66.35: 1
3.72.33.250: 1
13.40.204.215: 1
3.10.17.66: 1
34.223.68.251: 1
3.12.217.171: 1
16.171.51.216: 1
3.15.35.220: 1
3.101.176.163: 1
3.101.200.93: 1
3.101.176.234: 1
3.98.87.79: 1
Deleting gateways for site 'https://www.showmyip.com'.
Deleted 10 endpoints with for site 'https://www.showmyip.com'.
Ge0rg3 commented
Hi @Raychanan, thanks for the issue. This is working as expected, each "endpoint" only specifies which region the IPs will come from. Through API Gateway there is no way to purposefully reuse the same address twice
Raychanan commented
Great! This will save me a lot of scraping work! Thanks!