prestonp/scrape-goat

:goat: api for scraping urls with css selectors

Scrape Goat

API for scraping urls with css selectors

API

GET /?url=[url]&selector=[selector]

Returns JSON response containing

• status - the reponse status code
• results - the scraped html or values
• url - the requested url
• selector - the requested selector

Example

curl "http://localhost:3000/?url=http://www.nytimes.com/&selector=h2.story-heading%20a"

{
  status: 200,
  results: [
    "3 Americans Recount Attack on High-Speed Train to France",
    "3 Who Stopped Attack Were Boyhood Friends",
    "Profile Emerges of Suspect",
    "Ohio Bill Would Ban Abortion if Down Syndrome Is Reason",
    "As Polls and Fans Speak, Trump Shows Staying Power",
    "Biden, Considering White House Bid, Meets With Warren",
    ...
  ],
  url: "http://www.nytimes.com/",
  selector: "h2.story-heading a"
}

Live demo for my h8rs.

License

MIT