Immoscout with ScrapingAnt: Try a data-center request first before using residential proxies
tomburgert opened this issue · 4 comments
Hi,
first of all, thank you for your amazing work. I have a small feature request: would it be possible when scraping immoscout with the help of ScrapingAnt to try to use a data-center proxy call before using residential proxies? The current call consumes 250 credits, which means even with paying a monthly plan of 19$, only 400 calls would be allowed per month. Considering the very short reaction times that are needed to find a flat (for popular cities emails should be sent within 5 minutes) roughly 13 calls per day are just not enough. By using data-center proxies there would be chance of making 25x more calls. Otherwise, with the current consumption of credits, using fredy for immoscout crawling would not help a lot, unless they are able to spend 250$ a month for the service of ScrapingAnt.
Best,
Tom
Yep I'm planning to actually rework some parts around the immoscout scraping. I'll keep you posted
I talked to the founder of scraping ant. They have improved the detection rate on their end and I added this retry.
676d488
I will now first try it with a "regular" proxy, then switch to residential if detected. Keep in mind, this is the case in like 99 % of all cases.
Thank you so much for your quick support!