Separating tests meta-data from the testing engine code

[fredck]

We have concerns about the size of dist/tests.json. Currently it is a 200+KB file. Add this to the 250+KB of quail.jquery.js and Quail is starting to become a half-mega monster.

The point is that just a small part of this file is required to run Quail tests. For instance, properties like “testability”, “title”, “description”, “guidelines” and “tags” are just meta-data attached to the test and are not relevant to run the test.

This means that, if you run 200 tests in a document and just one issue is found, still you have to download the meta-data for all 200 tests, even if you application may be interested about that single issue information only.

To make it worst, even localization information is available for all languages and all tests. It’s clear that this will not work as soon as the number of languages supported start to grow.

There should be a clear separation on the way Quail is distributed. At one side it has a engine to run tests. At the other end it provides information about tests (or issues) that is relevant to applications using Quail, to show it to the end user.

And additional drawback is that the “custom tests” scripts are included inside quail.jquery.js, which need to be loaded on page load, instead of being loaded on demand inside tests.json, which could be called tests.js and have the custom code included there as well.

So we have two proposals:

1. Move custom tests from quail.jquery.js to tests.js (former tests.json). This will make quail.jquery.js much smaller and tests.js much bigger.
2. Move the meta-data about tests out of the above tests.js, optionally. This will make this file much smaller.

The idea for point “2” is having configurations for grunt build that would define whether or not to have the meta-data outside. Actually, it could even define what we want inside and what outside.

The result would be directory with the following (e.g.):

/dist/
    /quail.jquery.js
    /tests.js
    /meta/
        /testid/
            /en/
                meta.json
            /nl/ (…)

So, basically, every test would have it’s own directly inside “meta”, with a dedicated meta.json available for each supported language.

Inside tests.js, a test could look like this:

"aAdjacentWithSameResourceShouldBeCombined”: {
    "type": "custom",
    "callback": "aAdjacentWithSameResourceShouldBeCombined"
}

While in meta.json, we would have this:

{
    "testability": 1,
    "title": "Adjacent links that point to the same location should be merged",
    "description": "Because many users of screen-readers use links to navigate the page, providing two links right next to eachother that point to the same location can be confusing. Try combining the links.",
    "guidelines": {
      "wcag": {
        "2.4.4": {
          "techniques": [
            "H2",
            "F89"
          ]
        },
        "2.4.9": {
          "techniques": [
            "F89"
          ]
        },
        "4.1.2": {
          "techniques": [
            "F89"
          ]
        }
      }
    },
    "tags": [
      "link",
      "content"
    ]
}

As said, we could even make this configurable, so one could decide, for example, to have “testability” and “tags” available inside tests.js, so filtering could be applied when running tests.

Finally, one could also decide to have everything together, just like it happens today.

I hope all this makes sense.

[jessebeach]

Totally agree. I just moved the unit tests to Karma/Mocha/Chai/Sinon this past weekend: #279

I'm now working to move the accessibility test tests from the custom QUnit test runner to a standard Selenium test platform. Once we're collecting tests through a config file, we won't need the munged-together JSON file any more.

1. Move custom tests from quail.jquery.js to tests.js (former tests.json). This will make quail.jquery.js much smaller and tests.js much bigger.
2. Move the meta-data about tests out of the above tests.js, optionally. This will make this file much smaller.

Yes and yes. quail.jquery.js must be broken up. I'm plowing towards this goal before the end of 2014.

[jessebeach]

Work is ongoing here in the selenium-testing branch. It will probably take me another 6 weeks to complete this work.

[WilcoFiers]

As discussed during last week's call, here is my proposal for this approach:

• Guidelines: Section 508 has moved to WCAG 2, so we can probably remove this for a next version of Quail. As for WCAG2, a different approach to testing for WCAG 2 conformance has been build for Quail. This supersedes the wcag2 value, so this too can be removed. This makes the entire 'guidelines' property redundant.
• Title/description should be moved to their own localisation files. We should use URIs for identifying the different assessments, and have the localisation files define the title/description that belongs to that URI.
• Testability is an important property of the assessments, so I'd suggest keeping that in.
• Tags should get their own mechanism. Rather then be build into the core, we should have a component for tags that defines a subset of the assessments to be tested. We'd give that it's own class and have each 'tag' defined in a separate configuration file

[jessebeach]

I pulled the metadata into each assessment file.

#387