HTTPArchive/data-pipeline

Issues

• Assess secondary page quality

  #127 opened a year ago by rviscomi
  2

• Improved documentation for the reports

  #114 opened a year ago by Themanwithoutaplan
  2

• Investigate views on `all` dataset

  #142 opened a year ago by tunetheweb
  8

• Investigate missing Top 1k home pages

  #222 opened a year ago by rviscomi
  13

• Create a process to backfill `all` tables from HARs older than March 2022

  #138 opened a year ago by giancarloaf
  0

• KeyError: 'wasm'

  #243 opened a year ago by rviscomi
  3

• Clean up `beam_bq_job_LOAD` partial tables

  #210 opened a year ago by rviscomi
  1

• Incomplete HTML response bodies

  #188 opened a year ago by rviscomi
  2

• Convert the `latest` dataset to views

  #141 opened a year ago by rviscomi
  16

• Crawlid to be continued after EOL of batching

  #113 opened a year ago by Themanwithoutaplan
  5

• Add report generation to workflow (move off of worker VM)

  #177 opened a year ago by giancarloaf
  2

• Split the custom metrics JSON into structured fields

  #262 opened a year ago by rviscomi
  15

• In future trim down custom metrics from `payload`?

  #269 opened a year ago by tunetheweb
  4

• Add `rank` field to `all.requests`

  #189 opened a year ago by rviscomi
  13

• Consider clustering `all.requests` table by `page` or `rank`

  #263 opened a year ago by tunetheweb
  4

• The new schema and cost concerns for users

  #149 opened a year ago by tunetheweb
  6

• Explore integrating with Cloudflare's Domain Intelligence API

  #143 opened a year ago by rviscomi
  4

• Create auto updating `sample_data` queries

  #150 opened a year ago by tunetheweb
  5

• Add monthly CWV Tech Report materialization query

  #237 opened a year ago by rviscomi
  7

• Dataflow job failure: crawl-combined-desktop-20240227-042336-753436z

  #245 opened a year ago by rviscomi
  1

• Error -3 while decompressing data: incorrect header check

  #244 opened a year ago by rviscomi
  2

• Summary statistics for BigQuery `all` tables

  #124 opened a year ago by giancarloaf
  1

• Summary data in `all.requests` issues

  #257 opened a year ago by tunetheweb
  2

• Pub/Sub Bottleneck

  #128 opened 2 years ago by pmeenan
  1

• Rerunning a workflow with existing BQ data doesn't skip its Dataflow job

  #208 opened 2 years ago by rviscomi
  1

• Investigate why the Dataflow jobs stall on the remaining HARs

  #160 opened 3 years ago by rviscomi
  1

• Account for redirects when selecting secondary page URL

  #139 opened 3 years ago by rviscomi
  2

• Combined data pipeline failed to process October dataset

  #152 opened 3 years ago by rviscomi
  0

• Add GitHub Action to build flex template and deploy workflow on merge to main branch

  #168 opened 3 years ago by giancarloaf
  0

• Set up mechanism for triggering the batch pipeline

  #120 opened 3 years ago by rviscomi
  1

• Run November BigQuery pipeline

  #155 opened 3 years ago by rviscomi
  9

• Too many pages in `summary_requests`

  #153 opened 3 years ago by rviscomi
  1

• Blink feature tables haven't been updated

  #148 opened 3 years ago by tunetheweb
  1

• Page summary reports contain duplicates

  #112 opened 3 years ago by Themanwithoutaplan
  4

• August "combined" tables contain duplicates

  #133 opened 3 years ago by rviscomi
  5

• Secondary pages marked as root pages in parsed CSS table

  #116 opened 3 years ago by rviscomi
  1

• Switch data pipeline defaults from streaming to batch

  #119 opened 3 years ago by rviscomi
  0

• Keep secondary page data separate from home page data (for now)

  #93 opened 3 years ago by rviscomi
  0

• Eliminate the need to run batch pipelines

  #92 opened 3 years ago by rviscomi
  0

• Clean up intermediary load job tables on BQ

  #117 opened 3 years ago by rviscomi
  2

• Exceeded rate limits: too many api requests per user per method for this user_method

  #103 opened 3 years ago by rviscomi
  4

• GoogleAPICallError: None POST https://bigquery.googleapis.com/bigquery/v2/[...]/insertAll Error 413 (Request Entity Too Large)!!1

  #110 opened 3 years ago by giancarloaf
  1

• Error: The Dataflow job may be impacted by insufficient Pub/Sub quota

  #102 opened 3 years ago by rviscomi
  3

• Investigate why the crawl stalled on the remaining 300k HARs in June 2022

  #94 opened 3 years ago by rviscomi
  2

• Prepare for the July 2022 crawl

  #98 opened 3 years ago by rviscomi
  1

• Renew HTTP Archive API Key

  #95 opened 3 years ago by rviscomi
  1

• Seeing some big changes in requests for June

  #84 opened 3 years ago by tunetheweb
  14

• Populate requestid field

  #86 opened 3 years ago by giancarloaf
  0

• Turn off the expiration for experimental tables on BigQuery

  #82 opened 3 years ago by rviscomi
  1

• Blink tables contain 20220512 data and not 20220501

  #83 opened 3 years ago by tunetheweb
  1