More Sites to Scrape
rpmullig opened this issue · 6 comments
I attempted to adjust the 'providers' in the settings.yaml, but I found a few that raised errors. The following would be great additions to impact the tool:
- 'hire.google'
- 'Angel.co'
- 'greenhouse.io'
- 'jobs.jobvite'
- 'workable'
Just adding to this list here, so it is captured in the right place: https://news.ycombinator.com/jobs
This seems like a decent source as well: https://remote.co/remote-jobs/
Before moving into development, I worked in Finance and found many venture capital firms that had websites with listings of jobs for their investments. I could find a few of the main ones and post here.
Is there any thoughts on inverting the dependency and make the scrapers pluggable? I'm thinking there is opportunity to define a clear interface/abstract class for people to implement their own scrapers. Later on, we could use the entrypoint mechanism to enable people to implement plugins in a separate package. In that way, the main JobFunnel framework can become more stable and anyone could create and maintain their own plugins as necessary as Python packages. Once in pypi, a requirements file could eventually look something like:
jobfunnel
jobfunnel-linkedin
jobfunnel-monster
...
@Josian I'd be interested to see this idea with a more fleshed-out example architecture, It might be a good idea for maintainability and as the bloat grows around special cases.
I could see that fitting into the current ABC design where we are essentially building Base objects and then specializing per-locale, per-provider.
If you're interested in taking this further, a design for the class stubs would be a good place to start the discussion.
#135 indeed India