sitespeedio/sitespeed.io

Comparison of metrics with previous runs

Closed this issue · 7 comments

anonymous73612 commented

Your question

Hello,

We are measuring Core web vitals on our website with sitespeed.io. It is hooked to our CI/CD, using Docker and Kubernetes.
We want to compare the metrics with the previous runs to verify if they have improved/degraded. Please suggest how this can be achieved. Is Graphite a good option here and we are not using Docker Compose?
We are only concerned about Core web vitals.

soulgalore commented

Hi @anonymous73612 it depends on how you run and what you want to accomplish. If you push the metrics to Graphite, you can use Grafana and alert on regressions. You can also use the performance budget (but then you will compare against a hard limit): https://www.sitespeed.io/documentation/sitespeed.io/performance-budget/

As a third alternative you can use the compare plugin https://www.sitespeed.io/documentation/sitespeed.io/compare/ - that can compare with last run or a benchmark run.

anonymous73612 commented

Thank you for your answer, we are already using the performance budget to compare against the limits set by Google.
I will try the compare plugin and see how it goes.

anonymous73612 commented

I tried the compare plugin and it says 'You should use 20+ iterations to get statistical significant data'. Is it not possible to use less number of iterations?

anonymous73612 commented

I have python3 installed and working but compare plugin is using python?
I am getting this error:

HomePageTest.js_US:err: [2024-03-26 17:09:49] WARN: You should use 20+ iterations to get statistical significant data
[2024-03-26 17:10:31] ERROR: Error: Command failed with ENOENT: python /Users/goyalsahil/Desktop/FEPerfTests/hdms-frontend-performance-tests/frontend-performance-tests/node_modules/sitespeed.io/lib/plugins/compare/statistical.py
spawn python ENOENT
spawn python ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
[2024-03-26 17:10:31] ERROR:

anonymous73612 commented

Okay, the compare plugin works for me now thanks, Is there a way of comparing Google web vitals and not comparing everything?

soulgalore commented

Hi @anonymous73612 not in the HTML. I use the metrics in Graphite, so I alert on the ones I'm interested in.

anonymous73612 commented

Thanks @soulgalore for the quick response