flathunter only works for flats ;)

[pcace]

Hi,,

this is more a question than an issue. Is there any way to use flathunter also for other searches on immobilienscout24.de?
So i realized, that it workes perfectly now on "Wohnung mieten" or "Haus mieten" but then it does not work on "Haus kaufen"
(example search string: https://www.immobilienscout24.de/Suche/de/berlin/berlin/haus-kaufen?enteredFrom=one_step_search)
python3 flathunter.py [2020/06/08 16:27:04|config.py |INFO ]: Using config /scripts/flathunter/config.yaml [2020/06/08 16:27:04|idmaintainer.py |INFO ]: already processed: 10 Traceback (most recent call last): File "flathunter.py", line 91, in <module> main() File "flathunter.py", line 87, in main launch_flat_hunt(config) File "flathunter.py", line 46, in launch_flat_hunt hunter.hunt_flats() File "/scripts/flathunter/flathunter/hunter.py", line 38, in hunt_flats results = searcher.get_results(url) File "/scripts/flathunter/flathunter/crawl_immobilienscout.py", line 33, in get_results 0].text) ValueError: invalid literal for int() with base 10: '1.228 '

neither on "Grundstück kaufen"
(search string: https://www.immobilienscout24.de/Suche/de/grundstueck-kaufen?plotarea=-5000.0&price=-10000.0&pricetype=buy&enteredFrom=result_list)

python3 flathunter.py [2020/06/08 16:28:51|config.py |INFO ]: Using config /scripts/flathunter/config.yaml [2020/06/08 16:28:51|idmaintainer.py |INFO ]: already processed: 10 Traceback (most recent call last): File "flathunter.py", line 91, in <module> main() File "flathunter.py", line 87, in main launch_flat_hunt(config) File "flathunter.py", line 46, in launch_flat_hunt hunter.hunt_flats() File "/scripts/flathunter/flathunter/hunter.py", line 38, in hunt_flats results = searcher.get_results(url) File "/scripts/flathunter/flathunter/crawl_immobilienscout.py", line 37, in get_results entries = self.extract_data(soup) File "/scripts/flathunter/flathunter/crawl_immobilienscout.py", line 101, in extract_data entries.append(details) UnboundLocalError: local variable 'details' referenced before assignment

would there be a quick fix for that, or is this just not planned or wanted?

Thank you so much in advance!

Cheers

[codders]

Hey there pcace,

There's no reason why that couldn't work. The reason it breaks right now is probably just that the HTML on those pages is structurally different to the HTML on the rental pages.

The system is designed to be extensible. It would be quite easy to create a new crawler to handle those URLs. Right now, that isn't my priority, and I think the other Flathunters are renting flats, but if you wanted some hand-holding to create your own parser for those pages I could support you with that.

Thanks again,

Arthur

[pcace]

Hi,
thanks again for your superfast reply ;)! i guess i would take the efford and look into it. what could possibly go wrong ;).
Since i am a total python noob, it will take me some time, but i guess its worth learning.

Maybe you could point me to the right spots to look?

Thank you anyways!

Cheers!

[codders]

Sure - it'll mostly be copy and paste anyway :)

If you look at flathunter/crawl_immobilienscount.py, that's the existing ImmoScout crawler. You can copy that and make your own (e.g. flathunter/crawl_immobilienscout_haeuser.py), and then do the same for the test (test/test_crawl_immobilienscout_hauser.py).

You can edit the test case and change the test URL. If you can make the test past (pytest test/test_crawl_immobilienscout_hauser.py), you're almost ready to integrate.

Let me know how you get on!

Arthur

[codders]

Not so much activity here. I've created a project to track this: https://github.com/flathunters/flathunter/projects/8 .