Re-enable some browser-testing CI

Question

Re-enable some browser-testing CI

drzraf opened this issue 4 years ago · 16 comments

https://www.browserstack.com ($30/mo) karma-plugin
https://testingbot.com/ (free for FLOSS) karma-plugin
https://saucelabs.com/pricing ($20/mo) karma-plugin
https://www.lambdatest.com/ (free-tiers 60 mins/mo, then $15/mo)

One issue with SauceLabs is that the karma plugin apparently got outdated (because protocol evolves, etc...)

Now there is also this karma-webdriver-launcher plugins which aims to be more standard via webdriver (which wap them all).
It's said to support SauceLabs and even advised by lambdatest.
Even more interesting (from the above pricing), TestingBot says it's supported by webdrive.io.

@AidasK ?

Answer 1 · 2020-10-16T14:11:20.000Z

Great research, any of these is ok for me as long as it works. 👍

Answer 2 · 2020-12-31T20:14:20.000Z

I shared the access to testingBot.
I could not find "public test page" so far
Travis variable may need to be https://docs.travis-ci.com/user/environment-variables/#defining-variables-in-repository-settings

By the way, it seems that you didn't actually removed sauceLabs : I see it running on v3 commits.
But then, do you know why it stopped being shown (eg: at #321) ?
I (maybe wrongly) thought you purposefully disabled it.

Answer 3 · 2021-01-04T07:51:54.000Z

You can write to: aidaskk@gmail.com

I have tried adding these to travis, though travis is currently crashing. It says I don't have enough permissions on https://travis-ci.org/ and once I have moved to travis-ci.com, it does not work at all.

Answer 4 · 2021-01-04T07:55:51.000Z

I have given you admin access to this repo, maybe this repo will appear on your travis-ci account

Answer 5 · 2021-01-04T08:03:56.000Z

Thank you for supporting this @drzraf !

Answer 6 · 2021-01-04T08:07:28.000Z

* https://www.browserstack.com ($30/mo) [karma-plugin](https://github.com/karma-runner/karma-browserstack-launcher)

* https://testingbot.com/ (free for FLOSS) [karma-plugin](https://github.com/testingbot/karma-testingbot-example)

* https://saucelabs.com/pricing ($20/mo) [karma-plugin](https://github.com/karma-runner/karma-sauce-launcher)

* https://www.lambdatest.com/ (free-tiers 60 mins/mo, then $15/mo)
One issue with SauceLabs is that the karma plugin apparently got outdated (because protocol evolves, etc...)

Now there is also this karma-webdriver-launcher plugins which aims to be more standard via webdriver (which wap them all).
It's said to support SauceLabs and even advised by lambdatest.
Even more interesting (from the above pricing), TestingBot says it's supported by webdrive.io.

@AidasK ?

Actually Saucelabs is for free for opensource projects (https://saucelabs.com/open-sauce); If you are looking for alternatives let's try to see if they offer similar plans

Answer 7 · 2021-01-04T08:31:45.000Z

@evilaliv3 good to see you back. @drzraf is migrating flow.js to v3 version with some breaking changes. He is mostly redoing each callback and event in promise based approach

Answer 8 · 2021-01-04T19:09:27.000Z

@evilaliv3 TestingLabs is free for FLOSS too (I added you there too btw).
I didn't intended at all to change something that worked (SauceLabs) but wondered why I was not seeing the testsuite being run for each PR anymore.

If it's "just" that CI wouldn't run for PR done against the v3 branch then we may not need to switch (unless you still find other compelling argument in favor). If we decide to switch, the PR is almost ready.

@AidasK : I'm not used to GitHub CI settings/travis integration but I'll have a look.

Answer 9 · 2021-01-04T20:26:47.000Z

I'm on it, let me check and i will be back to you

Answer 10 · 2021-01-04T20:36:52.000Z

@drzraf : Actually saucelabs accepts all the builds we make and we don't have conditional code in relation to the branch name.

I suspect the bug could lie in the addition of these lines:


  if (config.sauceLabs && (!config.sauceLabs.username || !config.sauceLabs.accessKey)) {
    console.log('Undefined sauce username/accessKey.');
    process.exit(1)
  }

I cannot retest now but i would suggest to remove those and give it an other try

Answer 11 · 2021-01-06T15:23:29.000Z

@evilaliv3 : You're right.

Github handover to Travis.
.travis-ci.yml defines SAUCE_*
Gruntfile.js takes from process.env.SAUCE_* into sauceLabs (grunt.initConfig)
normally passed over to karma.conf.js
But karma.conf.js (karma-sauce-launcher) works without having to define this (native support for environment variables) and the change you mentioned kept it from running.

The two possible path are:

#327
#325

If SauceLabs is simply fixed (#327): Do you still think OP (karma-plugin /support) is a compelling arguments for switching or not? Are there others?

Answer 12 · 2021-09-16T15:41:03.000Z

@AidasK, could you have a look at travis-ci in order to enable it for v3?
Here is what I see (in my case, flow.js isn't available to integrate with)

Answer 13 · 2021-09-17T07:03:20.000Z

I couldn’t activate this. I tried reinstalling the app, but it does not work anyway. I have made you an owner of flow.js, so you should have an access now to try this out yourself. Hope this helps

Answer 14 · 2021-09-17T16:25:04.000Z

I sent their support a message. BTW when enabling Travis-CI for "All repositories", I only see "fusty-flow.js" and "ngx-flow".
What could be particular to these two? And what is "fusty-flow.js"?

Answer 15 · 2021-09-17T18:25:37.000Z

Great! It must be an issue at Travis side. fusty-flow.js is a fallback for flow.js for old browsers such as IE8. It’s no longer relevant.

Answer 16 · 2021-09-28T15:33:07.000Z

I was able to do without Travis at all (Only saucelabs and codeclimate). Will do a PR soon.
But codeclimate experiences the same problem as Travis regarding detecting available organizations (according to Travis support, it's an ongoing Github API problem)