Run each set of tests in separate iframe

[hakubo]

This would make karma also superb for integration (and such) tests.
As they can easily pollute global variables or fixtures.

This would be awesome to have files variable in config file to accept something like:

files = [
    ['file.js', 'file.spec.js'],
    ['lib.js', 'lib.spec.js']
]

or an object

files = {
    'fileTest': ['file.js', 'file.spec.js'],
    'libTest': ['lib.js', 'lib.spec.js']
}

so you could name each set

and each array would be set of files to be loaded in a separate iframe
so you would not have to worry about polluting environment
and just focus on integration tests itself

[bitwiseman]

👍
I have to do this on my own to shim my integration tests, so this seems like a fine idea.

[huyz]

I'm also interested in how to groups of tests in isolation without fear of interference.

[vojtajina]

Do not do global state and you gonna be fine ;-)

This is related to sharding (for huge test suites - one could run parts on
different machines to make it faster). It's not a high priority right now,
but I like this idea.

On Tue, Mar 19, 2013 at 11:51 PM, Huy Z notifications@github.com wrote:

I'm also interested in how to groups of tests in isolation without fear of
interference.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/412#issuecomment-15160826
.

[hakubo]

I like the idea of not doing global states! But life is not that beautiful :)

[huyz]

It's not just a problem with global states.
If you're using RequireJS, then all the modules that have already been loaded could have state modified by prior tests.

[dignifiedquire]

What about adding a reset/reload mechanism? That way there is a way to ensure you have a clean slate but we don't need to manage multiple iframes.

On 21.03.2013, at 10:12, Huy Z notifications@github.com wrote:

It's not just a problem with global states.
If you're using RequireJS, then all the modules that have already been loaded could have state modified by prior tests.

—
Reply to this email directly or view it on GitHub.

[huyz]

Sure. Although I'm not sure how it would work, given the example for main-test.js at http://karma-runner.github.com/0.8/plus/RequireJS.html, which looks to me as if the requireJS state wraps the karma runner.

[hakubo]

Also having that done for separate iframes is a step towards sharding that @vojtajina mentioned.

[vojtajina]

@huyz What you described IS a global state. It might be not accessible/visible, but it's still a global state.

[huyz]

@vojtajina point taken, thanks.

[hakubo]

Any Idea when this could be done ?
If not, could someone show me some pointers on where to look ?

[jperl]

You could also run your tests using uitest.js. It loads up your application in an iframe.

[hakubo]

This looks good!
I'll definitely look into it.

[vojtajina]

It's not high priority now, so not sure when this gonna happen...

On Thu, Mar 28, 2013 at 2:17 AM, Jakub Olek notifications@github.comwrote:

This looks good!
I'll definitely look into it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/412#issuecomment-15575219
.

[vojtajina]

I'm closing this as there is already an issue for sharding - splitting a big suite into multiple chunks, so that it is possible to parallelize the run (run the same suite on multiple browsers). #439

I don't think we need a fresh iframe per spec. It would make the run much slower. I still believe fighting global state is better ;-) So far I saw a few apps that wanted this so badly (per spec iframe) and it was always possible to refactor the code to not need it.

[fredericoschardong]

@hakubo did you end up coding this feature? I need this for a plugging I am working on, it would be good to not start from scratch. Thanks!

[hakubo]

@fredericoschardong Unfortunately no. I have changed my stack in the meantime.

[bonitao]

+1, the proposed grouping syntax looks very nice.

[timdp]

While it's not an ideal solution, I've written a Karma plugin that injects an iframe with a postMessage bridge into Mocha tests. It's called karma-mocha-iframes and there's also a demo.

[sabberworm]

I’ve tried to generify @timdp’s solution. The result is karma-iframes. It’s the same idea but should work with any adapter, not just mocha.

[timdp]

@sabberworm Are you sure it's the same idea? From the readme, it sounds like you're running each test file in a separate iframe. My version runs each individual (Mocha) test in its own iframe to prevent each test from tainting the context for subsequent ones.

[bashmish]

This might be really helpful when testing custom elements (web components). For now there is only one single registry and no way to unregister an element. Though there is this idea to have scoped registries which does not have a concrete proposal yet.

This old discussion about the possibility to unregister didn't go further than pointing out that there are test frameworks which already run tests in iframes, which is the web-component-tester. But web-component-tester is also very limited and only allows running individual suites in iframes, not individual tests. I created a repo to show this behavior. Looks like the above mentioned karma-iframes is similar to the WCT behavior.

I can understand that this use case is not trivial. But maybe it is time to reconsider and reopen the issue at least to explore possible use-cases further.

[lancedikson]

I'm currently having the same situation with custom elements (Web Components) testing and struggling with setting up integration and unit tests, where each component can be registered only once and there's no way to unregister it. It produces very specific workarounds after all. Let's try to call active maintainers to re-check this issue and maybe re-open it because the problem is getting more and more common due to the growing popularity of WebComponents. @johnjbarton 🙏

[lancedikson]

One more hint for those, who suffer from the issue and don't want to use karma-iframes because of their drawbacks: I came up with a wrapping decorator for @customElement():

import { customElement } from 'lit-element';

function customElementDecorator(tagName:string): any {
  const registeredElement = window.customElements.get(tagName);

  if (typeof registeredElement === 'undefined') {
    return customElement(tagName);
  }
  return () => {};
}

@customElementDecorator('my-component')
class MyComponent {
// ...
}

Then you can place this decorator to some optional requirement in webpack, for example, and use it only during testing, like this:

const legitCustomElement = (NODE_ENV === 'testing') ? customElementDecorator : customElement;
export { legitCustomElement as customElement };

This workaround seems working for now, let's see how it goes when we have more tests. I hope there will not be any race conditions with registering elements from different parts of the code.