/cloudquery

Turn any website to serverless API (support SPA!)

Primary LanguageJavaScriptMIT LicenseMIT

CloudQuery

Turn any website to serverless API (with SPA support!)

Try online: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/

API

Example url: https://2uvlp0y13b.execute-api.us-west-2.amazonaws.com/staging/query?url=https://news.ycombinator.com/best&selectors=*:nth-child(1) > * > *:nth-child(2) > *:nth-child(2) > :nth-child(1),:nth-child(3) *:nth-child(1) > *:nth-child(3) > *:nth-child(1)

Query params:

  • url: url of the webpage you want to fetch
  • selectors: css selectors describing the elements interests you

Returns:

  • innerText: element innerText
  • href: element href

Run locally

npm i
npm start
# open http://localhost:3000

How to deploy your own CloudQuery to AWS lambda

1. Configuration

  • update profile(AWS Credential Profiles) in up.json to use your own aws account
  • update rateLimit in config.json to set your own rate limit (Default: 5 request/hour)

2. Deploy

  1. npm install
  2. Install up
  3. Input up in terminal and hit enter :)
  4. up url to see the API baseURL of your CloudQuery

How to Contribute

  1. Install nodemon and parcel locally
  2. Start backend by npm run local
  3. Start frontend by 
        cd frontend
    npm install
    npm start
  4. The backend and frontend server will both auto restart when seeing code change
  5. After adding new feature or fixing bugs, execute npm run build in frontend folder, frontend code will be built and copied to public folder.

Develop online

You can run CloudQuery in a free online dev environment using Gitpod:

Open in Gitpod

Thanks

  • up for deploying serverless API to AWS with ease
  • serverless-chrome for running chrome on AWS lambda
  • finder for making it easy to select elements on webpage