RunPage should not use iframes for testing

[Krinkle]

Right now the client runner is the RunPage. There the browser periodically asks for new runs to run. And when it gets one, it add a temporary iframe to the document that is pointed to wherever the test suite is located (can be on a different origin).

The test framework includes our inject.js file which we use to "phone home" (through window.parent, since origin restrictions only apply when a window accesses a child frame, not the other way around).

Over time <iframe> has causes more and more trouble frames. We should consider finding a better method for this. An environment that is more reliable. Ideally in a stand-alone window. But just setting window.location isn't going to do it for us, because we need to keep track of the progress. And when the tests are finished, we need to be able to send a (pretty big) javascript object back up to the RunPage, which then submits it through AJAX to the API (again we can't cross-domain AJAX from the child frame, but we can from the parent window).

Moments ago on IRC I was bringing up the idea of using window.open() to create a popup and run the tests there. As far as controllability goes, this shouldn't be an issue. In JavaScript one can close windows that are opened by the same script (right?). But then we need an alternative way to send the javascript objects (and it needs to work in old browsers as well).

I'm not sure, but I don't think a window.parent like relationship exists through window.open() (even though the original window does somehow get the authority to close it) that we can use to do window.parent.SWARM.someMethod();.

[scottgonzalez]

window.opener

[Krinkle]

Can opener?

Beer opener?

---

Anyway, I heard of the (non-standard but widely implemented) window.opener (MDN) property, but it has different restrictions.

Contrary to frames, the relationship from the child window to the parent is restricted by the origin policy:

[jzaefferer]

We're running tests on the same origin as testswarm itself, so that's what you should be testing, not opening swarm from MDN.

And the same applies to iframes. You can open them anywhere, but you can only control their content when its on the same origin. Right?

[Krinkle]

Yes, we ("jquery") are running them from the same domain (by adding a non-standard directory to the root of the TestSwarm install and exporting all the test suites in there before submission). But that is (afaik) by no means the required situation for TestSwarm.

The exact example I did earlier (opening swarm from another domain) is backwards indeed, but that doesn't matter – the principle is the same. I could just as well have done it the other way around (opening another site from swarm). Either way, window.open() does not allow the child to access window.opener (when the origin doesn't match).

I tested it just now and it is indeed the same for frames:

It turns out I wrote it wrong yesterday as well, which caused the confusion. We do indeed need access to the parent from the - potentially cross-domain - child (not the other way around). But, we don't need access to the entire window object. (e.g. not window.parent.SWARM.someMethod), what we need is access to postMessage. And that (duh!) has an exception to this rule (both for frames and windows):

There is still the matter of browsers that don't support postMessage, but we already have a fallback to that (we build a <form>, that posts cross-domain to a special receiver in TestSwarm. And that receiver, being on the same domain as the parent, can then access the appropriate method directly through window.parent.SWARM.someMethod).

With frames that works (we already do that), I wasn't sure if it would also work with a window, but it turns out it does. The reason I thought it wouldn't is because I didn't think the window.opener property would be preserved when navigating to a different location. But that works OK (at least in Chrome, lets hope it works in IE6 too):

[Krinkle]

So, I just pushed the branch, and it is all working as expected (haven't found any issues yet). Except that there's one blocking issue that I don't know how to solve.

"What?", you ask? The friggin' popup blocker (duh!).

• Chrome 19:
  • Popup: Blocked.
    It is just the window being suppressed. The window is opened in a hidden process and interacting with it works fine.
  • Focus: Medium
    No focus since it is hidden, but focus is not an issue in Chrome.

  • PASSED

• Firefox 12:
  • Popup: Blocked.
    Completely prevented (return of window.open is null).
    Once whitelisted, it works fine.
  • Focus (once enabled): Medium
    No focus since new windows go into a tab, and new tabs are not focused, but focus is not an issue in Firefox.

  • FAILED

• Safari 5.1:
  • Popup: Blocked.
    Completely prevented (return of window.open is undefined).
    No visual notification of it being prevented, no whitelist in preferences. The feature can only be turned off globally, from the preferences. Once disabled, it opens actual new windows and everything is fine.
  • Focus (once enabled): Good
    Opens a new window and that window is focussed

  • FAILED

• Opera 11.6
  • Popup: Bocked.
    Prevented but somewhat initialized (return of window.open is a Window object, with closed!=true, but the document is not parsed yet).
    Shows dialog to allow opening it, which works. But that first window lost its window.opener reference and window.close is inaccessible. Refreshing after allowing it works fine, but only for 1 round. The next run gets blocked again.
  • Focus (once enabled): Variable
    When manually allowed the new window goes into an immediately focussed tab. The next ones usually aren't focussed.

  • FAILED

• IE 9
  • Popup: Blocked.
    Completely prevented (return of window.open is null).
    Choosing "Options for this site> Always allow" makes it work. The new window goes into an immediately focussed tab. However it turns out IE only supports postMessage() for frames, not for windows. It completely lacks indicating this until you call it, it throws: SCRIPT16386: No such interface supported. Feature test for window.opener.postMessage works fine. We'll have to find a way to fallback to <form> POST-ing for all IE versions (instead of just for those without support for postMessage). See also msdn blogs.
  • Focus: Good

  • FAILED

(the '>' indicate the test result on a clean machine with the default browser settings after a cold boot (which is what BrowserStack will provide) - not the test result after interacting with whatever dialogs, resetting the (now failed) test run, and trying again).

[jzaefferer]

I saw you mentioning something in IRC. Is that realistic? Otherwise we should just kill that one bad test in jQuery Core, if that's all we trying to address here. I think @dmethvin already pretty much agreed to that.

[Krinkle]

I am sure we can and am convinced we will eventually end up doing in-host page control and direct client-server communication, but I'm not sure I have time for it in the next few weeks. Lets keep it as a future feature.

[treasonx]

I was wondering if you could point me in the right direction to get an understanding of what troubles the iframes are causing?

I am currently developing a thirdparty javascript framework in my day job. And I do a lot of cross domain iframe communication. I am a little confused why you would need to open a new window? It seems like an iframe would be fine.

Right now in the test runner i built for my company I have a test iframe which communicates with its parent using easyXDM to support older browsers. http://easyxdm.net/ The test runner iframe communicates with its parent yielding progress and test results. You cannot call functions directly on the iframe itself but you can pass async messages.

[Krinkle]

Opening a new window isn't a solution either, because of popup blockers and what not. Initial research has shown it is not reliable to open a new window every now and then to run a test.

Iframes are still the way to go for now, and they're working fine (test suites for jQuery core and jQuery UI are passing fine). However there are some edge cases where Internet Explorer reports offsets wrong when run from within an iframe.

Running into this is fairly unlikely and it can be worked around.

For now we'll stick to iframes. In the long run we're likely switching to a system where the test run page becomes part of the navigational flow (as opposed to running it from a "run" page that will run the tests as children in iframes). In other words, redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).

Anyway, that's all future talk. Unless you're hitting a problem with iframes, I'd recommend just ignoring this for now.

[mikesherov]

We're running into this now on jquery UI as well. It seems like your latest comment is the way to go @Krinkle. Redirect from the "run" page to test page, which should then submit results and redirect back to the run page.

[mikesherov]

I can work on this issue now if the pull request would be accepted. Even though the practical case of "it breaks jquery and jquery ui" are present, I'm especially on board with the philosophical case of "your test environment should match your actual environment as closely as possible", and running in an IFRAME certainly isn't.

[Krinkle]

@mikesherov That depends on the approach you plan to take on the implementation. Feel free to post it here.

[mikesherov]

@Krinkle,well there are 2 issues, from my naive perspective...

1. navigation: this would be handled by the process you've described: redirecting to the test suite, running the test suite and submitting the results, and then (by the inject.js code that is inside the test runner) it will redirect back to an idle page where it will wait until the next run is sent to it (through a socket).
2. timeouts: 2 things here...
   1. inject.js now needs to be aware of the timeout, so it can post back to the swarm when a timeout happens. However, this doesn't cover the case of what happens when the test page breaks inject.js's ability to even report on the timeout.
   2. We'd need a cron that cleans up stalled runs after a while. This can be at the minute level of granularity, and can clean up on the next minute after the timeout.

I haven't yet done a deep dive of the code... are there other issues here that need to be covered other than what I've described above?

[staabm]

Just a note: hoe do you implement the redirection?

Javascript location? Meta redirect? Http location header?

In case you choose http location, make sure you use temporary redirects, no permanent (http 301 vs. 302). Permanent redirects will be cached by some browsers without asking the server for the actual location..

[Krinkle]

It would be JavaScript location. Because the user will be idling on the run page, and the polling (or socket push) will notify of a new run available. Redirecting from there will only redirect the asynchronous request, not the user viewport.

[jpvincent]

Hi

I am in an environment where I use testswarm to execute fonctional tests, because usual tool like Selenium do not exist. It's "pages" written for browsers shiped with TVs (HbbTVs). In half TVs (there are around 15 models in 2013), some tests fail because they are executed inside an iframe.
I would like to have the choice to execute them "fullscreen".

The last update to this issue was 10 months ago : do you have any plan to test this mode ?

[Krinkle]

@jpvincent If you know a way to trigger fullscreen from JavaScript, do that from inside your test page – not from TestSwarm. If you try it in TestSwarm, it would make the TestSwarm wrapper page fullscreen, not the client unit test page.

TestSwarm is responsible for running tests. The user is responsible for configuring browsers and pointing them to the swarm.

[Krinkle]

For unit testing (TestSwarm's primary purpose) iframes should not be a major inconvenience. jQuery Core, QUnit, and others are passing in it fine. We've had very few issues in all these years.

For UI testing, iframes can be more cumbersome. But I find in practice that when a functional library is doing heavy DOM interaction in a unit test means, it usually means one of four things:

• It's testing UI components with a unit test framework.
• It's unit testing a function by needlessly going through the DOM to trigger it – call functions directly.
• It's not testing actually user code at all – but needless confirming that firing events results in their callback being run. (Upstream does this for you already; Add integration tests instead e.g. using webdriver).
• It's jQuery.

But most importantly, the obvious alternative (a pop-up) is not a viable solution as outlined in previous comments (e.g. pop-up blocker). The only viable work-around is to rewrite the navigation code to use redirect management in form of a host page. I'm open to that but would recommend a separate issue for that. Or possibly a different framework instead of TestSwarm (e.g. Karma).