On 2024-07-03 we created a webinar with Evgeny Fomenko from iProxy. During the webinar Kameleo's CEO and Co-founder, Tamás Deák showcased how Kameleo can be used to scrape behind login walls, and bypass any anti-bot system. Here we share the code that was used during the demo.
To run the examples you will need to do some preparation. If you are stuck, we can help you.
To run the following code examples, you need to run Kameleo.CLI on your machine.
During the demo we used iProxy's product as a proxy. The following values should be changed, so you can access and control your own proxy
- proxy_host
- proxy_port
- proxy_username
- proxy_password
- ip_renewal_url
To easily start scraping behind login wall, create new Kameleo profiles, and log in to an existing website with them. Keep them in Kameleo's workspace, and add ip:profile_name key:values to the profile store sqlite file. This way you will be able to load back the profiles later. Contact support if you have any issues.
Start app.py to run the example.
- Code will request a new IP
- Code will determine the IP address
- Code will search for any virtual browser profile that was previously used with the given IP
- Code will launch the found profile, and drive it with playwright
If IP from the pool was not used before, code will create a new virtual browser profile for that.
We really want you to succeed with this project. So please contact our support team if you have any issues. We will help you with the examples, and we will also provide you with prepared .kameleo profiles