HTTPArchive/tech-report-apis

Document proposed architecture

rviscomi opened this issue · 5 comments

rviscomi commented

Describe how we'd serve data from the API endpoint, extracting from BigQuery and updating at the end of each monthly crawl

rviscomi commented

httparchive.core_web_vitals.technologies is the source of truth and there will be one script for each of the API endpoints that query the data from the table

rviscomi commented

Hook into the automated pipeline to update API data on Firestore on a monthly basis: https://github.com/HTTPArchive/data-pipeline/blob/main/data-pipeline.workflows.yaml

Changing the API code itself will require a manual deployment script

rviscomi commented

API security: as needed implement CORS and/or API keys

rviscomi commented

Firestore

  • global instance?
  • lives in the project, serverless service
  • not like a VM instance
  • data stored in us-east
  • possible to put a CDN in front of API gateway
maceto commented

@rviscomi @tunetheweb I changed the region for the project to us-east4, which is the default for the project. Now Firestore database, API gateway, and cloud functions are all in the same region. It should work with the performance.

And looking for information about CDN in front of the API, I found this documentation https://cloud.google.com/api-gateway/docs/multi-region-deployment

I tried to test it but I don't have permission to create all necessary resources.