/cloudquery

Turn any website to serverless API (support SPA!)

Primary LanguageJavaScriptMIT LicenseMIT

CloudQuery

Turn any website to serverless API (with SPA support!)

Try online: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/

API

Example url: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/query?url=https://news.ycombinator.com/best&selectors=*:nth-child(1) > * > *:nth-child(2) > *:nth-child(2) > :nth-child(1),:nth-child(3) *:nth-child(1) > *:nth-child(3) > *:nth-child(1)

Query params:

  • url: url of the webpage you want to fetch
  • selectors: css selectors describing the elements interests you

Returns:

  • innerText: element innerText
  • href: element href

Run locally

npm i
npm start

How to deploy your own CloudQuery to AWS lambda

1. Configuration

  • update profile(AWS Credential Profiles) in up.json to use your own aws account
  • update rateLimit in config.json to set your own rate limit (Default: 5 request/hour)

2. Deploy

  1. npm install
  2. install up
  3. Input up in terminal and hit enter :)
  4. up url to see the API baseURL of your CloudQuery

Contribute

Backend

vim app.js

Frontend

cd frontend
# develop
npm start
# build
npm run build

Thanks

  • up for deploying serverless API to AWS with ease
  • serverless-chrome for running chrome on AWS lambda
  • finder for making it easy to select elements on webpage