SiteSpeed test exits with 1 with suppressExitCode in place
Closed this issue · 5 comments
niskhakova-gitlab commented
Have you read the documentation?
- Yes, I've read the how to make a reproducable bug guide
- Yes, I've read the how to debug my script guide
URL
https://staging.gitlab.com/explore
What are you trying to accomplish
When using SiteSpeed v31.1.x version, exit code is 1 with budget.suppressExitCode in place. When running the same command using v31.0.x - exit code is 0.
Might be related to the recent change in v31.1 - #4044 but needs to be confirmed. Thanks in advance!
What browser did you use?
Chrome
How to reproduce
## sha256:5f212ca4ae23de9888f00ab089471277b0f93d41a572a95658d6b60d71abaac8
export SITESPEED_IMAGE=sitespeedio/sitespeed.io:latest
# budget.json https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/blob/main/budget.json?ref_type=heads, config.json https://gitlab.com/gitlab-org/quality/performance-sitespeed/-/blob/main/config.json?ref_type=heads
docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io $SITESPEED_IMAGE --config config.json --budget.suppressExitCode --budget.configPath budget.json https://staging.gitlab.com/explore -n 1
echo $?
1
# Using older image
export SITESPEED_IMAGE=sitespeedio/sitespeed.io:31.0.1
docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io $SITESPEED_IMAGE --config config.json --budget.suppressExitCode --budget.configPath budget.json https://staging.gitlab.com/explore -n 1
echo $?
0Log output
https://staging.gitlab.com/explore -n 1
Google Chrome 120.0.6099.109
Mozilla Firefox 121.0
Microsoft Edge 120.0.2210.77
[2024-01-16 12:38:14] INFO: Versions OS: linux 6.2.0-1018-gcp nodejs: v20.9.0 sitespeed.io: 31.1.1 browsertime: 21.0.0 coach: 8.0.2
[2024-01-16 12:38:14] INFO: Running tests using Chrome - 1 iteration(s)
[2024-01-16 12:38:15] INFO: Use Chrome trace categories: -*,disabled-by-default-lighthouse,v8,v8.execute,blink.user_timing,devtools.timeline,disabled-by-default-devtools.timeline,disabled-by-default-devtools.timeline.stack
[2024-01-16 12:38:15] INFO: Testing url https://staging.gitlab.com/explore iteration 1
[2024-01-16 12:38:43] INFO: Take after page complete check screenshot
[2024-01-16 12:38:45] INFO: Take cumulative layout shift screenshot
[2024-01-16 12:38:46] INFO: Take largest contentful paint screenshot
[2024-01-16 12:38:50] INFO: Use the visual metrics portable script
[2024-01-16 12:38:50] INFO: Get visual metrics from the video
[2024-01-16 12:38:58] INFO: https://staging.gitlab.com/explore 29 requests, TTFB: 697ms, firstPaint: 1.30s, firstVisualChange: 1.27s, FCP: 1.30s, DOMContentLoaded: 1.83s, LCP: 1.30s, CLS: 0.1819, TBT: 449ms, CPUBenchmark: 64ms, Load: 1.90s, speedIndex: 1.30s, visualComplete85: 1.27s, lastVisualChange: 1.90s
[2024-01-16 12:38:59] INFO: Failing budget First Contentful Paint for https://staging.gitlab.com/explore with value 1.296 s max limit 1.000 s
[2024-01-16 12:38:59] INFO: Failing budget Total Blocking Time for https://staging.gitlab.com/explore with value 449 ms max limit 300 ms
[2024-01-16 12:38:59] INFO: Failing budget Cumulative Layout Shift for https://staging.gitlab.com/explore with value 0.182 max limit 0.1
[2024-01-16 12:38:59] INFO: Budget: 3 working, 3 failing tests and 0 errors
[2024-01-16 12:39:01] INFO: HTML stored in /sitespeed.io/sitespeed-result/staging.gitlab.com/2024-01-16-12-38-14
1 page analysed for https://staging.gitlab.com/exp... (1 run, Chrome/desktop/[object Object])
Score / Metric Median
------------- ------
! First Paint 1.296 s
√ First Contentful Paint 1.296 s
Fully Loaded 2.305 s
Page Load Time 1.902 s
√ Largest Contentful Paint 1.296 s
! First Visual Change 1.266 s
Last Visual Change 1.900 s
√ Speed Index 1.305 s
! Visual Readiness 634 ms
TTFB 697 ms
Cumulative Layout Shift 0.182
✗ Total Duration of Long Tasks 664 ms
! Number of Long Tasks 2
! Total Blocking Time 449 ms
✗ Max Potential FID 499 ms
√ Coach Overall Score 92
√ Coach Best Practice Score 91
√ Coach Privacy Score 98
! Coach Performance Score 89
√ Total Requests 29
Image Requests 2
✗ Total Transfer Size 1.5 MB
HTML Transfer Size 12.4 KB
JavaScript Transfer Size 538.9 KB
✗ JavaScript Content Size 1.7 MB
CSS Transfer Size 133.2 KB
Image Transfer Size 6.1 KBsoulgalore commented
That looks like something I broke, sorry. I didn't remember we had that suppress functionality. Let me fix that for the next release.
soulgalore commented
I released 31.2.0 today, hopefully that will fix the problem, please let me know when you have tried it.
niskhakova-gitlab commented
@soulgalore many thanks for the quick turnaround on this! I tried the latest version and unfortunately exit code is 1 with --budget.suppressExitCode in place. Please let me know if something's needed from my side.
docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:latest --config config.json --budget.suppressExitCode --budget.configPath budget.json https://staging.gitlab.com/explore -n 1
Google Chrome 120.0.6099.109
Mozilla Firefox 121.0
Microsoft Edge 120.0.2210.77
[2024-01-18 10:22:13] INFO: Versions OS: linux 6.2.0-1018-gcp nodejs: v20.9.0 sitespeed.io: 31.2.0 browsertime: 21.0.0 coach: 8.0.2
[2024-01-18 10:22:13] INFO: Running tests using Chrome - 1 iteration(s)
[2024-01-18 10:22:14] INFO: Use Chrome trace categories: -*,disabled-by-default-lighthouse,v8,v8.execute,blink.user_timing,devtools.timeline,disabled-by-default-devtools.timeline,disabled-by-default-devtools.timeline.stack
[2024-01-18 10:22:14] INFO: Testing url https://staging.gitlab.com/explore iteration 1
[2024-01-18 10:22:42] INFO: Take after page complete check screenshot
[2024-01-18 10:22:44] INFO: Take cumulative layout shift screenshot
[2024-01-18 10:22:46] INFO: Take largest contentful paint screenshot
[2024-01-18 10:22:50] INFO: Use the visual metrics portable script
[2024-01-18 10:22:50] INFO: Get visual metrics from the video
[2024-01-18 10:23:00] INFO: https://staging.gitlab.com/explore 28 requests, TTFB: 769ms, firstPaint: 1.36s, firstVisualChange: 1.37s, FCP: 1.36s, DOMContentLoaded: 1.81s, LCP: 1.36s, CLS: 0.0063, TBT: 0ms, CPUBenchmark: 64ms, Load: 1.95s, speedIndex: 1.52s, visualComplete85: 2.03s, lastVisualChange: 2.03s
[2024-01-18 10:23:00] INFO: Failing budget First Contentful Paint for https://staging.gitlab.com/explore with value 1.363 s max limit 1.000 s
[2024-01-18 10:23:00] INFO: Budget: 4 working, 1 failing tests and 0 errors
[2024-01-18 10:23:03] INFO: HTML stored in /sitespeed.io/sitespeed-result/staging.gitlab.com/2024-01-18-10-22-13
1 page analysed for https://staging.gitlab.com/exp... (1 run, Chrome/desktop/[object Object])
Score / Metric Median
------------- ------
! First Paint 1.363 s
√ First Contentful Paint 1.363 s
Fully Loaded 2.177 s
Page Load Time 1.950 s
√ Largest Contentful Paint 1.363 s
! First Visual Change 1.366 s
Last Visual Change 2.033 s
√ Speed Index 1.517 s
! Visual Readiness 667 ms
TTFB 769 ms
Cumulative Layout Shift 0.006
✗ Total Duration of Long Tasks 686 ms
✗ Number of Long Tasks 3
√ Total Blocking Time 0 ms
√ Max Potential FID 0 ms
√ Coach Overall Score 93
√ Coach Best Practice Score 99
√ Coach Privacy Score 98
! Coach Performance Score 89
√ Total Requests 28
JavaScript Requests 12
CSS Requests 5
✗ Total Transfer Size 1.5 MB
HTML Transfer Size 12.4 KB
JavaScript Transfer Size 539.3 KB
✗ JavaScript Content Size 1.7 MB
CSS Transfer Size 133.5 KB
Image Transfer Size 1.1 KB
echo $?
1soulgalore commented
Sorry new try with 31.2.1. This time I could verify that it works for me.
niskhakova-gitlab commented
@soulgalore thank you very much for the quick fix, truly appreciate the help on this! 🙇 Confirmed that it works 🎉