/scrapoxy

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers πŸ•ΈοΈ. It also smartly handles traffic routing πŸ”€ to minimize bans and increase success rates πŸš€.

Primary LanguageTypeScriptMIT LicenseMIT

Scrapoxy

What is Scrapoxy?

Scrapoxy is a super proxy aggregator, allowing you to manage all proxies in one place 🎯, rather than spreading it across multiple scrapers πŸ•ΈοΈ.

It also smartly handles traffic routing πŸ”€ to minimize bans and increase success rates πŸš€.


Scrapoxy

πŸš€πŸš€ GO TO SCRAPOXY.IO FOR MORE INFORMATION! πŸš€πŸš€

Features

☁️ Datacenter Providers with easy installation ☁️

Scrapoxy supports many datacenter providers like AWS, Azure, or GCP.

It installs a proxy image on each datacenter, helping the quick launch of a proxy instance. Traffic is routed to proxy instances to provide many IP addresses.

Scrapoxy handles the startup/shutdown of proxy instances to rotate IP addresses effectively.

🌐 Proxy Services 🌐

Scrapoxy supports many proxy services like Rayobyte, IPRoyal or Zyte.

It connects to these services and uses a variety of parameters such as country or OS type, to create a diversity of proxies.

πŸ’» Hardware materials πŸ’»

Scrapoxy supports many 4G proxy farms hardware types like Proxidize.

It uses their APIs to handle IP rotation on 4G networks.

πŸ“œ Free Proxy Lists πŸ“œ

Scrapoxy supports lists of HTTP/HTTPS proxies and SOCKS4/SOCKS5 proxies.

It takes care of testing their connectivity to aggregate them into the proxy pool.

⏰ Timeout free ⏰

Scrapoxy only routes traffic to online proxies.

This feature is useful with residential proxies. Sometimes, proxies may be too slow or inactive. Scrapoxy detects these offline nodes and excludes them from the proxy pool.

πŸ”„ Auto-Rotate proxies πŸ”„

Scrapoxy automatically changes IP addresses at regular intervals.

Scrapers can have thousands of IP addresses without managing proxy rotation.

πŸƒ Auto-Scale proxies πŸƒ

Scrapoxy monitors incoming traffic and automatically scales the number of proxies according to your needs.

It also reduces proxy count to minimize your costs.

πŸͺ Sticky sessions on Browser πŸͺ

Scrapoxy can keep the same IP address for a scraping session, even for browsers.

It includes HTTP requests/responses interception mechanism to inject a session cookie, ensuring continuity of the IP address throughout the browser session.

🚨 Ban management 🚨

Scrapoxy injects the name of the proxy into the HTTP responses.

When a scraper detects that a ban has occurred, it can notify Scrapoxy to remove the proxy from the pool.

πŸ“‘ Traffic interception πŸ“‘

Scrapoxy intercepts HTTP requests/responses to modify headers, keeping consistency in your scraping stack. It can add session cookies or specific headers like user-agent.

πŸ“Š Traffic monitoring πŸ“Š

Scrapoxy measures incoming and outgoing traffic to provide an overview of your scraping session.

It tracks metrics such as the number of requests, active proxy count, requests per proxy, and more.

🌍 Coverage monitoring 🌍

Scrapoxy displays the geographic coverage of your proxies to better understand the global distribution of your proxies.

πŸš€ Easy-to-use and production-ready πŸš€

Scrapoxy is suitable for both beginners and experts.

It can be started in seconds using Docker, or be deployed in a complex, distributed environment with Kubernetes.

πŸ”“ Free and Open Source πŸ”“

And of course, Scrapoxy remains free and open source, under the MIT license.

I simply ask you to give me credit if you redistribute or use it in a project πŸ™Œ.

A warm thank-you message is appreciated as well πŸ˜ƒπŸ™.

Documentation

More information on scrapoxy.io.

Contributors

Contributors

Want to contribute? Check out the guide!

Here is my contact on Linkedin Badge

Sponsorship

Scrapoxy is an open-source project. The project is free for users, but it does come with costs for me.

I invest significant time and resources into maintaining and improving this project, covering expenses for hosting, promotion, and more.

If you appreciate the value Scrapoxy provides and wish to support its continued development, discuss new features, access the roadmap, or receive professional support, please consider becoming a sponsor!

Your support would greatly contribute to the project's sustainability and growth:

Licence

See The MIT License (MIT)

Acknowledgements

I would like to thank all the contributors to the project and the open-source community for their support.

Follow-up

Discord Docker NPM

Star History Chart