austinoboyle/scrape-linkedin-selenium

job without description raise an exception

jeromebon opened this issue · 4 comments

For jobs with a position_elements but no description there might be no '.pv-entity__description', pos['description'] is then None wich led to the following Traceback:

File "/.../scrape_linkedin/Profile.py", line 154, in to_dict
    info = super(Profile, self).to_dict()
File "/.../scrape_linkedin/ResultsObject.py", line 13, in to_dict
    return dict(zip(self.attributes, vals))
File "/.../scrape_linkedin/ResultsObject.py", line 12, in <lambda>
    vals = map(lambda attr: getattr(self, attr), self.attributes)
File "/.../scrape_linkedin/Profile.py", line 82, in experiences
    jobs = list(map(get_job_info, jobs))
File "/.../scrape_linkedin/utils.py", line 141, in get_job_info
    pos['description'] = pos['description'].replace(
AttributeError: 'NoneType' object has no attribute 'replace'

The same case is handled by a try catch when there is no position_elements (l.157)

Instead of a try/catch, pos['description'] could be replaced by pos.get('description', ''), but the result would be '' instead of None

I will do a PR on saturday if there is none before

A PR would be great, otherwise if you send me a link to a Profile with the issue I'll fix 'er up.

One example: /in/eva-hugaerts-0799881a

pos['description'] = pos['description'].replace(

AttributeError: 'NoneType' object has no attribute 'replace'

        if pos['description']: #check if empty before replace string
            pos['description'] = pos['description'].replace(
            'See less\n', '').replace('... See more', '').strip()