javascript metric in many cases empty
fbuecklers opened this issue · 9 comments
Hi,
I have noticed that the javascript custom metric get an improvement in #42 , which now also collect statistics over the server generated HTML.
That's really cool since that was something we have missed beforehand.
However, it seems that the JavaScript metric is, for example, on the url https://www.decathlon.de/ just empty.
SELECT
JSON_EXTRACT_SCALAR(payload, '$._javascript') AS javascript,
FROM
`httparchive.pages.2022_06_01_*`
where url like 'https://www.decathlon.de%'
limit 1
If have quickly validated that finding, and it looks like that this not unique to that url.
SELECT
count(*) as records,
count(JSON_EXTRACT_SCALAR(payload, '$._javascript')) AS javascript,
FROM
`httparchive.pages.2022_06_01_*`
Shows that the metric is on ~2M cases null .
Also, the _wpt_bodies metric reports some errors in that cases.
{
"log": [
{
"type": "error",
"context": "anchors",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "title",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "meta_description",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "hreflangs",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "headings",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "structured_data",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "raw_html",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "canonicals",
"exception": {},
"message": "Cannot read properties of null (reading '0')"
},
{
"type": "error",
"context": "robots",
"exception": {},
"message": "Cannot read properties of null (reading '0')"
},
{
"type": "error",
"context": "visible_words",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "general",
"message": "Failed to create the custom metrics object: Cannot read properties of null (reading 'length')",
"exception": {}
}
]
}
It looks like that the underlying issue is that the magic $WPT_REQUESTS is maybe null?
We see a similar issue on several other domains where we are
working with. All of them are quite large E-Commerce companies in Germany / EU. Not sure but that can of cause bias statistics which you may collect over that metrics.
It would be cool if someone can take a look into this or maybe give me a pointer how I can help to find the underlying issue,
Thanks in advance
There were some fixes in September to add bodies from netlog requests (service worker and a few others) which also added requests to the list of requests that come from netlog instead of dev tools and they may not have all of the same fields.
It looks like the JS metric probably doesn't react well if fields are missing.
i.e.
let scripts = $WPT_BODIES
.filter((request) => request.type == "Script")
.map((file) => {
return { url: file.url, body: file.response_body };
});What will that do if 'type' isn't available on the request? At a minimum we should probably skip any records that don't have the fields we're interested in but for bonus points, there could be a pre-pass that populates 'type' from the sec-fetch-dest request header if it is present.
CC: @kevinfarrugia in case he has any insights on this custom metric
The issue is related to the custom metric, most likely the change in #41.
The root cause of the issue is because some scripts do not have a body and the custom metric is throwing an unhandled exception on:
let sourcemapURLs = scripts
.map((n) => {
if (n) {
let url = n.body?.match(sourcemapRegex)?.[1];
This happens in cases when the response does not include a body, for example a redirect (and possibly a 404). Take a look at Request 90 https://fast.smarketer.de/api/js/fast.js on https://decathlon.de as an example (WPT: https://webpagetest.httparchive.org/result/220928_FZ_D/1/details/#waterfall_view_step1)
Nice that was fast! Thanks 👍
If I understand the fix correctly it will Not fix the wpt-body metric. Does the metric have a similar issue?
@fbuecklers can you help me understand the issue with _wpt_bodies or where you are seeing the error logs please (link to WPT if possible)? From what I see, it shouldn't be related to the JavaScript custom metric though.
Actually, I have found that in the database as well:
SELECT
JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') AS wptBodies,
FROM
`httparchive.pages.2022_09_01_*`
where url like 'https://www.decathlon.de%'
limit 1
Will actually return this:
{
"log": [
{
"type": "error",
"context": "anchors",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "title",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "meta_description",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "hreflangs",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "headings",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "structured_data",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "raw_html",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "canonicals",
"exception": {},
"message": "Cannot read properties of null (reading '0')"
},
{
"type": "error",
"context": "robots",
"exception": {},
"message": "Cannot read properties of null (reading '0')"
},
{
"type": "error",
"context": "visible_words",
"exception": {},
"message": "Cannot read properties of null (reading 'length')"
},
{
"type": "error",
"context": "general",
"message": "Failed to create the custom metrics object: Cannot read properties of null (reading 'length')",
"exception": {}
}
]
}
Also some other metrics seems to be broken as well:
"_robots_meta": {
"error": "TypeError: Cannot read properties of null (reading 'filter')"
}
"_structured-data": "{\"structured_data\":{\"type\":\"error\",\"context\":\"structured_data\",\"exception\":{},\"message\":\"Cannot read properties of null (reading 'length')\"},\"log\":[{\"type\":\"error\",\"context\":\"structured_data\",\"exception\":{},\"message\":\"Cannot read properties of null (reading 'length')\"}]}",
"_valid-head": {
"error": "TypeError: Cannot read properties of null (reading '0')"
},
Hm, have further looked into that, and it looks like that this was not the case in the httparchive.pages.2022_06_01_ table. So maybe it breaks recently though the changes which @pmeenan mentioned.
@kevinfarrugia To add some numbers here:
The following query gives the results regarding the different metric errors:
SELECT
count(*) as records,
countif(JSON_EXTRACT_SCALAR(payload, '$._javascript') is null) AS javascriptError,
countif(JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') like '%Cannot read properties of null%') AS wptBodiesError,
countif(JSON_EXTRACT_SCALAR(payload, '$._robots_meta.error') like '%TypeError: Cannot read properties%') AS robotsMetaError,
countif(JSON_EXTRACT_SCALAR(payload, '$._structured-data') like '%Cannot read properties%') AS structedDataError,
countif(JSON_EXTRACT_SCALAR(payload, '$._valid-head.error') like '%TypeError: Cannot read properties%') AS validHeadError,
FROM
`httparchive.pages.2022_06_01_*`Results in:
[{
"records": "13371217",
"javascriptError": "1910134",
"wptBodiesError": "4078",
"robotsMetaError": "188203",
"structedDataError": "16",
"validHeadError": "125579"
}]With the except of the javascript metric, the numbers looks quite unobtrusive.
However, if I run the same query on the 2022_09_01 dataset I get the following result:
SELECT
count(*) as records,
countif(JSON_EXTRACT_SCALAR(payload, '$._javascript') is null) AS javascriptError,
countif(JSON_EXTRACT_SCALAR(payload, '$._wpt_bodies') like '%Cannot read properties of null%') AS wptBodiesError,
countif(JSON_EXTRACT_SCALAR(payload, '$._robots_meta.error') like '%TypeError: Cannot read properties%') AS robotsMetaError,
countif(JSON_EXTRACT_SCALAR(payload, '$._structured-data') like '%Cannot read properties%') AS structedDataError,
countif(JSON_EXTRACT_SCALAR(payload, '$._valid-head.error') like '%TypeError: Cannot read properties%') AS validHeadError,
FROM
`httparchive.pages.2022_09_01_*`[{
"records": "25244732",
"javascriptError": "7342080",
"wptBodiesError": "4681181",
"robotsMetaError": "4928074",
"structedDataError": "4675306",
"validHeadError": "5387378"
}]So in around 1/5 of all tests the different metrics raise an exception.
Should I open a new issue for it?
Thanks @fbuecklers yes that looks serious and worth investigating. The dataset did grow by about 2x over this time period, but the error rate seems to have increased disproportionately.