Unknown error after the program runs for a while

[addieg]

Hi there,

I am using your script, and after it runs for a while (searching around 150 zips), it throws an unknown error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (336, 77). Other element would receive the click:

...

(Session info: chrome=64.0.3282.186)

It did this three times already -- any idea what might be going on?
Many thanks,
Andreea

[ChrisMuir]

Hi, thanks for taking the time to file an issue on this. Could you copy/paste the full traceback? To diagnose, I'll need to know where in the code this exception is being thrown.

[addieg]

Traceback (most recent call last):
File "C:\Users\ageor\Dropbox\Documents\mit\assortment_problems\Zillow\get_data_BosSfDenver.py", line 4, in
execfile("C:\Users\ageor\Dropbox\Documents\mit\assortment_problems\Zillow\zillow_runfile.py")
File "C:\Users\ageor\Dropbox\Documents\mit\assortment_problems\Zillow\zillow_runfile.py", line 72, in
if zl.enter_search_term(driver, term):
File "C:\Users\ageor\Dropbox\Documents\mit\assortment_problems\Zillow\zillow_functions.py", line 100, in enter_search_term
button.click()
File "C:\Users\ageor\Anaconda2\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\ageor\Anaconda2\lib\site-packages\selenium\webdriver\remote\webelement.py", line 628, in _execute
return self._parent.execute(command, params)
File "C:\Users\ageor\Anaconda2\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Users\ageor\Anaconda2\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (336, 77). Other element would receive the click:
< div class="z-callout-body" >...< /div >
(Session info: chrome=64.0.3282.186)
(Driver info: chromedriver=2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT 10.0.16299 x86_64)

[ChrisMuir]

Thanks. So it's throwing the error when the code attempts to click the "search" button within function enter_search_term(). Looking through the full page source of the search page, I don't see any page element with class z-callout-body.

This error is typically caused by either 1) target element is physically covered by another element, or 2) target element is outside the current page (page just needs to be scrolled up or down). When the error is thrown, is the search bar scrolled out of view, or is there anything blocking the search bar (pop-up dialog box or something)?

[addieg]

Got it, thanks. I am not sure what happened when I got the error, but I made sure to avoid this, and now it seems to work.
On a different note, I now got a "UnicodeEncodeError: 'ascii' codec can't encode character u'\xae' " error.

Just wondering -- do you have any suggestions for adding some save points in the code? So you don't end up loosing everything you scarped so far?

Thanks -- appreciate it!

[addieg]

I changed the writing command to: pd.DataFrame(output_data, columns = columns).to_csv(file_name, index = False, encoding='utf-8'), and now it works. in case you want to add the encoding='utf-8' to your script.

[ChrisMuir]

I like the idea of writing the data to file periodically while the scraper is running, I think I'll work that change in.

And yeah specifying utf-8 in the call to to_csv also makes good sense. If you want to submit a pull request for that easy fix, it'd be much appreciated. If not, no worries, I'll get to it soon.

I'm going to open a new issue for each of these.

Thanks a bunch Andreea!