webscraper-bot

Discord web scraping bot used to scrape websites with dynamic content and send notifications when there is a new item.

Instructions

1.Adding bot to your server

Simply click the link: https://discord.com/api/oauth2/authorize?client_id=1007981622005596171&permissions=2147485696&scope=bot%20applications.commands

2.Adding new scraper job

Use /create-job command and fill in the form:

name - name of the job
url - url that the bot will scrape
selector - selector for an a tag with link to element (examples below)
interval - interval of the job in minutes (min 1 minute)
active? - default true, if the job should be active right away (you can always enable/disable job with [enable/disable]-job commands)
channel? - defualt channel where the command was run which channel should be messaged when the new item appear
clean? - default true, if the query params should be ignored (essential for some sites like Ebay)

After that you can do basic CRUD opeartions on jobs with commands like /list-jobs /update-job /delete-job...

3.Waiting...

After that you simply wait for the job run and it will send message when new items are found:

You can also run the job manually with /run-job. Note that the first run will get all the elements that will be on the webiste.

How to get selector

The selector is a querySelectorAll string that the bot uses to get unique a tags linking to items you want to scrape. To verify if your selector is working you can run document.querySelectorAll(<your selector>) in browser console and check if the function returns items you want to scrape.

Examples of selectors

Ebay: .srp-river-results .s-item__image a
Otodom: [data-cy=listing-item-link]
Olx: [data-cy=l-card] > a

jjablonski-it/webscraper-bot