ruippeixotog/scala-scraper

Too many redirects occurred trying to load URL

dloshkarev opened this issue · 3 comments

dloshkarev commented

I'm trying to parse url https://shop.rivegauche.ru/newstore/ru/%D0%91%D1%80%D0%B5%D0%BD%D0%B4%D1%8B/%D0%92%D1%81%D0%B5-%D0%91%D1%80%D0%B5%D0%BD%D0%B4%D1%8B/GUERLAIN/Guerlain-Meteorites-Gold-Light-The-Christmas-Collection/p/873725
But got java.io.IOException: Too many redirects occurred trying to load URL. Also I tried to parse same url using jsoup (1.11.2) lib for java and it was successful.
I guess the problem in method org.jsoup.helper.HttpConnection.Response#createConnection, because it have some differences in comparison with java method

ruippeixotog commented

Hi @dloshkarev! Can you share a minimal example of what you're doing in order for me to reproduce the problem?

dloshkarev commented

I added an example to https://github.com/dloshkarev/web-parser
Just run Main object for reproducing

ruippeixotog commented

Hi @dloshkarev, that's a strange bug indeed. I couldn't reproduce it before because it works correctly on the master branch, which already depends on a newer version of jsoup. It was probably a bug on their side, I suppose.

I have just released scala-scraper 2.1.0. Let me know if you still have problems with that version. Thanks for submitting the issue!