A set of jq scripts used to extract useful data from WebPageTest (WPT).
This repo is a follow up to this blog post I wrote about extracting certain data from the WebPageTest API using jq, which can then be used for other means.
I've created this repo to allow myself and others to quickly extract certain information from a WebPageTest API JSON blob (mainly as in CSV format), that will allow the data to be easily plotted in graphing tools. This gives people the ability to save / edit / modify graphs very quickly for presentations, blog posts etc.
In all the examples below I have used Google Sheets to plot the graphs. The example spreadsheet can be found here.
Description:
Extract DNS, Connect, SSL, timing information from multiple runs to a CSV for easy graphing.
Run data used:
All runs.
Sample graph:
Usage:
# local file
./connection-metrics.jq wpt-json-blob.json > connection-metrics.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./connection-metrics.jq > connection-metrics.csvDescription:
In this example we extract TTFB, First Contentful Paint, Start Render, DOM Complete , and Fully Loaded timing metrics as an example as to how to do this with other metrics.
Run data used:
All runs.
Sample graph:
Usage:
# local file
./extract-other-run-metrics.jq wpt-json-blob.json > extract-other-run-metrics.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./extract-other-run-metrics.jq > extract-other-run-metrics.csvDescription:
In this example we extract TTFB, First Contentful Paint, Start Render, DOM Complete , and Fully Loaded timing metrics as an example as to how to do this with other metrics. In this version we filter the data to remove any obvious errors.
Run data used:
All runs.
Sample graph:
Usage:
# local file
./extract-other-run-metrics-filtered.jq wpt-json-blob.json > extract-other-run-metrics-filtered.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./extract-other-run-metrics-filtered.jq > extract-other-run-metrics-filtered.csvDescription:
In this example we extract the cumulative layout shift information across n number of runs to a CSV file then plotted in Google Sheets.
Run data used:
All runs.
Sample graph:
Usage:
# local file
./cls-across-runs.jq wpt-json-blob.json > cls-across-runs.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./cls-across-runs.jq > cls-across-runs.csvDescription:
Here we replicate the WebPageTest requests pie chart which shows the percentage of types of requests (e.g. HTML, CSS etc)
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-mime-requests.jq wpt-json-blob.json > breakdown-mime-requests.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-mime-requests.jq > breakdown-mime-requests.csvDescription:
Here we replicate the WebPageTest bytes pie chart which shows the percentage of bytes per request type (e.g. HTML, CSS etc)
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-mime-bytes.jq wpt-json-blob.json > breakdown-mime-bytes.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-mime-bytes.jq > breakdown-mime-bytes.csvDescription:
Here we replicate the WebPageTest requests pie chart which shows the percentage of requests by domain.
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-domain-requests.jq wpt-json-blob.json > breakdown-domain-requests.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-domain-requests.jq > breakdown-domain-requests.csvDescription:
Here we replicate the WebPageTest bytes pie chart which shows the percentage of bytes per domain.
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-domain-bytes.jq wpt-json-blob.json > breakdown-domain-bytes.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-domain-bytes.jq > breakdown-domain-bytes.csvDescription:
Here we replicate the WebPageTest pie chart displaying the processing time (ms) for each of the categories.
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-processing-categories.jq wpt-json-blob.json > breakdown-processing-categories.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-processing-categories.jq > breakdown-processing-categories.csvDescription:
Here we replicate the WebPageTest pie chart displaying the processing time (ms) for each of the events.
NOTE: There is a limitation with pie charts in Google Sheets, where you can't conditionally colour code depending on the category a metric is within (e.g. Layout, Painting etc). The category classification has been included in the data should it be needed.
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-processing-events.jq wpt-json-blob.json > breakdown-processing-events.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-processing-events.jq > breakdown-processing-events.csvDescription:
Here we replicate the WebPageTest pie chart displaying the time (ms) for each of the categories (including idle time).
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-time-categories.jq wpt-json-blob.json > breakdown-time-categories.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-time-categories.jq > breakdown-time-categories.csvDescription:
Here we replicate the WebPageTest pie chart displaying the time (ms) for each of the events (including idle time).
NOTE: There is a limitation with pie charts in Google Sheets, where you can't conditionally colour code depending on the category a metric is within (e.g. Layout, Painting etc). The category classification has been included in the data should it be needed.
Run data used:
Median run.
Sample graph:
Usage:
# local file
./breakdown-time-events.jq wpt-json-blob.json > breakdown-time-events.csv
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./breakdown-time-events.jq > breakdown-time-events.csvDescription:
Here we want to compare the number of requests that were made of a particular asset type between tests (think the compare view on WebPageTest). This script pulls out the 2 sets of data ready to be plotted in a graphing tool.
Run data used:
Median run from each test.
Sample graph:
# local files
./compare-number-requests.jq wpt-json-blob-before.json wpt-json-blob-after.json > compare-number-requests.csvDescription:
Here we want to compare the number of bytes that were made of a particular asset type between tests (think the compare view on WebPageTest). This script pulls out the 2 sets of data ready to be plotted in a graphing tool.
Run data used:
Median run from each test.
Sample graph:
# local files
./compare-number-bytes.jq wpt-json-blob-before.json wpt-json-blob-after.json > compare-number-bytes.csvDescription:
Have jq walk over the WPT API response and generate a basic text based schema for easier searchability and overall structure familiarisation.
Data used:
Examines the whole JSON structure.
Sample response:
fvonly
successfulFVRuns
average
average/firstView
average/firstView/minify_total
average/firstView/responses_200
average/firstView/testStartOffset
average/firstView/bytesOut
average/firstView/gzip_savings
average/firstView/requestsFull
average/firstView/start_epoch
average/firstView/cpu.MarkLoad
average/firstView/connections
average/firstView/cpu.EventDispatch
...
Usage:
# local file
./generate-schema.jq wpt-json-blob.json > generate-schema.txt
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./generate-schema.jq > generate-schema.txtDescription:
The inner data structure for average, standardDeviation, median, and individual run numbers are all very similar. So what types of data is contained at this level? This script examines the inner data structure and outputs it as a JSON object for further examination.
Run data used:
Median run.
Sample response:
{
"Colordepth": "number",
"Dpi": "string",
"Images": "string",
"LastInteractive": "number",
"LayoutShifts": "array",
...
}Usage:
# local file
./inner-structure.jq wpt-json-blob.json > inner-structure.json
# curl
curl -s https://www.webpagetest.org/jsonResult.php?test=[WPT_TEST_ID]&normalizekeys=1 | ./inner-structure.jq > inner-structure.jsonUnless stated otherwise, the codebase is released under the MIT License. This covers both the codebase and any sample code in the documentation.