Test streamlining/browser support for W3C and other tests
brettz9 opened this issue · 6 comments
Although we have a recent addition of W3C-adapted tests for Node, I'd like us to:
- Node tests
- Use an automated process for converting the original browser-based W3C tests to Node tests
- The latest version of
ws
is breaking our worker code; ought to investigate rather than just
keeping the last working version hard-coded
- In-browser tests
- Convert the other fake, IndexedDBMock, and old W3C tests we've added to work for the browser
- Directly incorporate the original browser-based W3C tests for direct browser testing (as individual files)
- Get all browser tests to be able to launch together from within the browser
and get W3C test server working (if nothing else than for WebIDL tests; our tests are now copying the source of theWebIDLParser.js
file directly, so no need to build)-
Modifying the beginning ofWe can instead useFileHandler
get_data
inweb-platform-tests/tools/wptserve/wptserve/handlers.py
to the snippet below works but waiting on web-platform-tests/wpt#5133 . Alternatively, there are the following changes (they have the disadvantage of adding.any.js
files as per ournpm run w3c-add-wrap
routine and duplicating work in runner but the runner has the advantage of adding polyfill links alongside non-polyfill ones and was the recommended approach): web-platform-tests/wpt#5133npm run wrap
even though it unfortunately messes up theweb-platform-tests/IndexedDB
directory, requiring a reset of those files.
-
- Although there are Grunt tasks for auto-deleting db files, the tests themselves should really do the deletions on clean-up, ensuring proper browser testing (and less dead weight)
- Move "Old W3C" tests to own repository just in case ever need again, but as new W3C is set up for Node and browser now, we shouldn't need them anymore
- Get
** iframe tests (opaque tests) using polyfill in browser testsinterfaces.worker.js
and any other worker or - Get new W3C tests working on rawgit.com, ideally optionally with a runner that can work statically (without Python).
- Headless browsers
- Directly incorporate the original browser-based W3C tests for (Nightmare) headless browser testing
- Add Nightmare headless testing of fake/indexeddbmock/mocha/qunit (already have Node and browser)
- Actual testing
- Grab the low-hanging fruit in getting failing tests to pass (Need to see why Chrome and Safari are failing where the Node implementation is not)
- Report any new issues that cannot be readily solved
Three long-term goals (which are not milestone 3 but which I'll track here for now):
- For our pre-existing tests, maybe also streamlining to choose either Mocha in place of QUnit, unless there is some compelling advantage for QUnit tests.
- We might also try to return our tests to the W3C tests unless they are specific to our implementation only.
- See the
todo
markers in code for minor lower priority issues not being tracked
- Snippet
if byte_ranges is None:
if (path.find('IndexedDB') > -1 and path.find('.htm') > -1):
shim = """
<script src='http://localhost:9999/dist/indexeddbshim-UnicodeIdentifiers.js'></script>
<script>
shimIndexedDB.__setConfig({
fullIDLSupport: true,
replaceNonIDBGlobals: true
});
shimIndexedDB.__useShim();
</script>
"""
fc = open(path, 'rb').read()
return fc.replace("<script", shim + "<script", 1)
else:
return open(path, 'rb')
** Could add the following to workers in theory (and in Safari 10, in the workers only, the old bug of needing to define indexedDB
in a closure as an alias to shimIndexedDB
was also present):
importScripts('http://localhost:9999/dist/indexeddbshim-noninvasive.js');
setGlobalVars(null, {
fullIDLSupport: true,
replaceNonIDBGlobals: true
});
shimIndexedDB.__setConnectionQueueOrigin();
shimIndexedDB.__useShim();
However, Safari apparently removed its support of IndexedDB from workers (as did Chrome) as per https://bugs.chromium.org/p/chromium/issues/detail?id=434740
Note that it is now possible to set up a robust testing environment for the latest W3C tests (along with our own good number of tests), both for Node and the browser.
The W3C test runner for the browser still has to be patched (I've submitted a couple PRs as per web-platform-tests/wpt#5133 (comment) ), but if you get the web-platform-tests
test environment working (a submodule of this repo), the tweaks are simple enough to do by hand pending any incorporation of the PRs.
We could really use help, with WebSQL-supporting browsers (modern Chrome (57) and Safari 10) attempting to fix the issues they have in failing those tests that are working in Node.js (the ones still failing in our Node implementation are pretty much unresolvable, particularly in the browser). Chrome would probably be best to start with as it is easier to test and it has fewer failures that are not present in Node.
The test runner is not as useful in Safari which chokes on even native testing, so in that case, it will be necessary to test the file one-by-one or by regex (adding .any.html
to the file names as discussed in the above-mentioned web-platform-tests
issue).
As far as iOS9, I intend to post shortly to #277 in this regard.
Here's the code I currently have for an alternative testharness runner, which I've placed at testharness_runner_polyfill.html
at the root of web-platform-tests
along with the above-mentioned patches:
<!DOCTYPE html>
<meta charset="utf-8" />
<link rel="shortcut icon" href="data:image/x-icon;," type="image/x-icon" />
<script src="http://localhost:9999/dist/indexeddbshim-noninvasive.js"></script>
<iframe id="ifr" style="width:100%; height: 680px;" src="http://web-platform.test:8000/tools/runner/index.html"></iframe>
<script>
var win = document.querySelector('#ifr').contentWindow;
win.addEventListener('load', function () {
var oldLink = win.VisualOutput.prototype.link;
win.VisualOutput.prototype.link = function(href) {
var link = oldLink.apply(this, arguments);
link.target = '_blank';
var linkPolyfill = link.cloneNode(true);
var suffix = '.any.html';
if (!(/\.any\.html$/).test(href)) {
linkPolyfill.href += suffix;
linkPolyfill.textContent = 'polyfill';
}
var span = document.createElement('span');
span.appendChild(link);
span.appendChild(document.createTextNode(' ('));
span.appendChild(linkPolyfill);
span.appendChild(document.createTextNode(')'));
return span;
};
win.testharness_properties = Object.assign({}, win.testharness_properties, {
callback: function (win) {
setGlobalVars(win, {
fullIDLSupport: true,
replaceNonIDBGlobals: true
});
win.shimIndexedDB.__useShim();
// Object.defineProperty(win.IDBTransaction, 'READ_WRITE', {configurable: true, value: 55555}); // Use with `/IndexedDB/historical.html` to confirm we're injecting
}
});
});
</script>
Also adding this to web-platform-tests
' .gitignore
is helpful:
# Polyfill-related additions
IndexedDB/*.any.js
IndexedDB/_indexeddbshim-loader.html
testharness_runner_polyfill.html
Nevermind the above instructions--just follow the instructions in docs/TESTING.md
@brettz9 - should I archive this repo, so that we can move to the new location ?
See my comment at #353 (comment) . Thanks!