JakeChampion/polyfill-library

CI is broken and it seems unlikely that Browserstack will be able to resolve this on their own.

romainmenke opened this issue · 11 comments

romainmenke commented

see : https://github.com/Financial-Times/polyfill-library/actions/runs/4501232331/jobs/8176234150

{
  error: javascript error: Error: Permission denied to access property "global_test_results"
  Build info: version: '3.2.0', revision: '8c03df6', time: '2017-03-02 09:34:51 -0800'
  System info: host: 'mac-207-254-53-192.browserstack.com', ip: 'fe80:0:0:0:108f:a82:d0fa:f85e%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.15.6', java.version: '1.8.0_51'
  Driver info: org.openqa.selenium.firefox.FirefoxDriver

I know that Browserstack is rolling out breaking changes to their platform.
This is a gradual roll out, older browser versions seem to update first.
I've already reached out to them and asked them to provide more info on this.
This is a slow and frustrating process because everything goes through non-technical support. It is impossible to contact an engineer and just get a direct answer on what is going on.

The most clear info I have gotten is that they are removing a lot of heuristics around the capabilities which need to be set. In the past it was fine to just request a session with a specific browser name and browser version. Now a lot more needs to be set, all driver versions, machine type, ... Without these the service is either flaky or plain broken. I don't know why these heuristics are being removed. It is very difficult to manually configure everything correctly for the large number of browsers and browser versions we test here.

One way I worked around this in other projects was by setting a flag to make sure the w3c standard protocol wasn't used. This seemed to trigger the old heuristics to kick in again. This no longer seems to work.

It is possible that they have moved on to a next step in these changes and that the current breakage is not caused by these capabilities, but by something else.

I don't currently have time to really debug this all over again and find out what the issues are this time and I also don't have time to hop on a call with Browserstack support and show that the issues are real.

Is there anyone with a Browserstack account and a setup of the CI for the polyfill-library that is willing to spend time on this?

mhassan1 commented

I tried replicating this from my machine against my own BrowserStack, but I wasn't able to. With command node ./test/polyfills/remotetest.js test-modified-only targeted director browser=firefox, tests seem to be passing:

 • Browser: firefox/38.0 Test config: targeted /          ✓ 1088 tests  65 seconds to complete
 • Browser: firefox/39.0 Test config: targeted /          ✓ 1082 tests  63 seconds to complete
 • Browser: firefox/40.0 Test config: targeted /          91/113
 • Browser: firefox/41.0 Test config: targeted /          ✓ 1068 tests  60 seconds to complete
 • Browser: firefox/42.0 Test config: targeted /          ✓ 947 tests  56 seconds to complete
 • Browser: firefox/43.0 Test config: targeted /          49/97
 • Browser: firefox/44.0 Test config: targeted /          75/95
 • Browser: firefox/45.0 Test config: targeted /          71/90
 • Browser: firefox/46.0 Test config: targeted /          ✓ 909 tests  51 seconds to complete
 • Browser: firefox/47.0 Test config: targeted /          51/88

Is there something else I should do to replicate?

romainmenke commented

Thank you for checking @mhassan1 🙇

This at least gives some indication that CI can still pass.
Let me try a few things and I'll get back to you!

romainmenke commented

Re-tested :

  • the failing job here in CI with FT Browserstack credentials
  • main branch locally with my credentials
  • the feature branch locally with my credentials

All are failing on exactly Firefox 45 with the same error.

The Firefox 45 session, was it running on Windows XP or a different OS, OS version?

mhassan1 commented

Windows XP. Here is a video. It times out near the end, but I think that's because I interrupted the test. Also, here is a video of 45.0 on Sierra.

Are you able to see the job run in the Automate tab?

mhassan1 commented

Hm. Maybe there is some difference between our plans; here is mine:
image

romainmenke commented

Personal plan :

Screenshot 2023-04-10 at 19 28 49

I don't know the plan details of FT

JakeChampion commented

@notlee and/or @wheresrhys - could you let us know the FT BrowserStack plans please?

wheresrhys commented

You mean this?

image

It says our percy is stopped (some allowance limits have been exceeded recently, so maybe we are being penalised) - could that be the issue?

romainmenke commented

Maybe something specific about sponsored accounts?

romainmenke commented

I've been reminding Browserstack of these issues regularly and got some good news today :

We have made some changes at our end. Please try running a few sessions and let us know if the issue persists. If found any issue, please share the sessionID here please.

I've rerun some tests here and CI seems fine again 🎉
Thank you everyone for checking things and for providing more info.

  • 🎉1