BehaviorVueTests is flaky (OSOE-662)

Question

BehaviorVueTests is flaky (OSOE-662)

Closed this issue a year ago · 18 comments

Piedone commented a year ago

Both test cases in BehaviorVueTests are flaky, but in a way that if they fail, they fail 3 times in a row.

QrCardScanShouldWorkAsync fails with "OpenQA.Selenium.NoSuchElementException: Unable to locate visible element: - By: CSS selector ".qr-card .full-name" - Search time: 10.009s"
QrCardScanShouldReportNotFoundAsync fails with "OpenQA.Selenium.NoSuchElementException: Unable to locate visible element: - By: CSS selector ".qr-card .message-error" - Search time: 10.01s"
For failing builds, see here (the rerun succeeded), here, and here.

Jira issue

Piedone commented a year ago

Awesome!

Answer 1 · 2023-07-26T23:24:06.000Z

And now this build is stuck in failing even after 3 reruns: https://github.com/Lombiq/Open-Source-Orchard-Core-Extensions/actions/runs/5673495019

Answer 2 · 2023-07-27T13:35:37.000Z

BTW this appears to be the issue with the standard (slow) GitHub-hosted runners, not the faster ones we use for PR builds.

Answer 3 · 2023-07-27T18:47:02.000Z

@DemeSzabolcs after you think you've fix this, please make sure to test it on the standard runners too.

Edit: I see you're already doing this, great!

Answer 4 · 2023-07-27T19:31:30.000Z

The problem here is that the video is not loading (even if we wait a lot) there is only a black screen. But I could only repro this on the slower runner. I will try to come up with something.

Answer 5 · 2023-07-27T19:59:10.000Z

It might have to do something with the software installed on the GitRunners runners vs the GitHub ones.

Answer 6 · 2023-07-31T22:23:10.000Z

What happens is, that the video in ModuleTests.BehaviorVueTests.QrCardScanShouldWorkAsync and ModuleTests.BehaviorVueTests.QrCardScanShouldReportNotFoundAsync in theory loads (I checked the stream and it's not null or zero and there is no error message), however on the failure screenshot the video is black. So it's loaded but not properly.

Waiting won't solve the issue, so I guess it's stuck.

Answer 7 · 2023-08-01T00:42:11.000Z

This sounds like some video codec is missing. BTW the passing rerun linked above was on the runner image 20230716.1.0 while all failings ones on 20230724.1.0. You can check out what change in these releases may have caused this. Note that 20230728.3.1 was just released, so there will be further changes.

Answer 8 · 2023-08-01T12:47:52.000Z

The most possible cause of this, is that the Node.js version is not compatible with the video codec.

Note that 20230728.3.1 was just released, so there will be further changes.

I checked it, but it didn't change the Node.js version, so the issue is still present.

I don't know what would be the best solution here.

We can temporarily ignore the tests and wait for a new runner, that has a compatible Node.js version (this would be luck dependent).
We can use the old runner, which seems overkill for just 2 tests (and we can't use that forever).
We can install a compatible codec (if there is one). But that will also result doing something in a workflow, for just 2 tests.
We can rework the fake video capture source and use a different format instead of mjpeg, however I'm not sure if that is possible.

Answer 9 · 2023-08-01T13:15:46.000Z

What does it mean that Node.js is incompatible with the video codec? How does Node.js have anything to do with video? What would make a new Node.js version compatible?

Answer 10 · 2023-08-01T15:36:48.000Z

I'm not sure yet, but that's the only thing, that I can find as a correlation between the failing and runner updates. I will do some research and get back.

Answer 11 · 2023-08-01T16:31:39.000Z

I updated my Node.js version from 18.15.0 to the same one as the current runner uses (18.17.0), and I could repro it locally:

So it's definitely the Node.js version.

Answer 12 · 2023-08-01T16:38:28.000Z

That's a good start, then. With a local repro you're in a better position to fix it, so please look into that. Using an old version of Node is not a suitable option. However, in the meantime, I've temporarily disabled the tests in dev.

BTW once you have specific details, @dministro might be able to give you pointers.

Answer 13 · 2023-08-03T17:56:43.000Z

Loading video from a webcam works, both just normally locally and during UI testing too. But I found this similar issue, it might be related: cypress-io/cypress#27424

Also, I downgraded my Node.js and it's still not working (I can repro it) locally. So it might be the Chrome version, as the issue I mentioned also states.

This should also explain why the runner updated caused it, since they updated Chrome too:

Answer 14 · 2023-08-08T16:57:09.000Z

After some painful Google Chrome downgrading (yes it's not that simple), I can confirm that Google Chrome itself is the problem. The video loads and works in previous versions (before 115) but not after it. I guess I will write a bug report.

Edit: I wrote a bug report, but it's not public, since you have to fill a form. I guess they have to approve it.

Answer 15 · 2023-08-24T19:48:52.000Z

A new Chrome version released (115 ->116). I tested it and it's still not working. I again tried y4m and mjpeg too. And just to make sure, I tested it outside the UI tests, by starting Chrome from a command line, but it's still not working, so I'm still researching things..

Answer 16 · 2023-09-15T00:00:29.000Z

I looked at this again. Updated everything to the latest, the results are the same.

I downloaded an example.y4m video and that one loaded correctly! So I will take a look at that one and I will try to convert our videos to that format.

Answer 17 · 2023-09-15T22:14:36.000Z

I solved this finally. For some reason -filter:v scale=480:-1 this was ruining the video. I got the original files (mp4) and looked at the ffmpeg documentation and used different methods to scale the video. Now it's working, I will also update the documentation. Mpjeg and y4m works too.