Testing entry point
lawnsea opened this issue · 12 comments
Summary
Problem
We want ingredients and pantries to be easy to test. Specifically, it should be easy to write both unit and functional tests, easy to run those tests, and easy to obtain coverage data.
This RFC proposes the ingredient interface to support these goals.
Solution Overview
Ingredients may define a test entry point (test.js) that runs the ingredient's unit and functional tests and reports TAP results. The test harness will ensure a Roux server is running before executing the test entry point. It will provide the URL for that server as well as any other configuration information via environment variables. The test harness will be able to execute the test entry points of multiple ingredients and report the aggregate results.
API
Test entry point
An ingredient MAY define a test entry point by including a file named test.js in the ingredient root. If present, the script MUST report its result by printing a Test Anything Protocol (TAP) test stream to stdout.
A test entry point MAY assume that a Roux server is running and WILL receive the root URL for its ingredient as an environment variable named ROUX_INGREDIENT_URL.
A test entry point MUST NOT assume that it is the only test entry point executing. The test runner MAY execute multiple entry points concurrently, so access to any shared resources MUST be coordinated via other means.
A test entry point MAY assume that a Roux server is running and WILL receive the root URL for its ingredient as an environment variable named ROUX_INGREDIENT_URL.
What happens in the case where it chooses to not assume this? Or should this read "A test entry point MUST...".
Its unclear if this RFC allows for running unit tests outside of a roux-server environment. Does it? I think it should. In other words, I should not have to run a roux-server to run unit tests, I should be able to run these on the command line and view the results in the terminal similar to how rmn's grunt test command works now.
Yeah, good question @lzilioli, that passage is kinda muddy.
What I meant is that the test entry point for an ingredient can always assume that a Roux server will be running, with its ingredient preview, assets, etc. available. This will be the case even when running the tests from the CLI. The entry point doesn't have to make use of the server, of course. This is so that a single test entry point can perform both unit and functional tests.
I guess another way to do this would be to only provide that environment variable if the server is running. That might make it easier to run unit tests more quickly.
I think we should have two separate entry points; one for functional tests and one for unit tests. I don't like the idea of enabling developers of tests to conflate the two since they generally test distinctly different things. The functional tests can assume there MUST be the environment variable while the unit tests MUST work in the absence of said environment variables.
- Luke Z.
On Oct 11, 2016, at 15:37, Lon Ingram notifications@github.com wrote:
Yeah, good question @lzilioli, that passage is kinda muddy.
What I meant is that the test entry point for an ingredient can always assume that a Roux server will be running, with its ingredient preview, assets, etc. available. This will be the case even when running the tests from the CLI. The entry point doesn't have to make use of the server, of course. This is so that a single test entry point can perform both unit and functional tests.
I guess another way to do this would be to only provide that environment variable if the server is running. That might make it easier to run unit tests more quickly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
I think we should have two separate entry points; one for functional tests and one for unit tests. I don't like the idea of enabling developers of tests to conflate the two since they generally test distinctly different things.
Users are still free - and encouraged - to separate their functional and unit tests. There are at least a couple other kinds of tests, however: visual regression and performance. I suspect we could come up with one or two more if we wanted.
Rather than have an entry point for each flavor of test, I prefer to have a single entry point that speaks a common API (TAP). As an aside, this is what I plan to propose for linting as well: a single entry point that will run whatever linting tasks the ingredient desires and report the results via a single consistent interface.
One note, if the spec is going to enforce details of the environment (eg: that a roux server is running):
A test entry point MAY assume that a Roux server is running and WILL receive the root URL for its ingredient as an environment variable named
ROUX_INGREDIENT_URL
It should probably also be able to control that environment. I'm thinking specifically about the version of roux server that is running; maybe that means using a peerDependency or similar in the pantry's package.json?
- Does this spec assume that all ingredients are client-side components and thus run the tests in a browser (or browser equivalent)?
- How will test-runner specific configuration be specified (or not) per ingredient/pantry?
- Is test execution time bound by a timeout or is it open-ended?
@knksmith57 I would expect the pantry to have a devDependency on roux-server. Are you thinking of a different configuration?
Does this spec assume that all ingredients are client-side components and thus run the tests in a browser (or browser equivalent)?
So, I think there's a place for unit tests that run in node instead of a browser. That said, I haven't really contemplated non-client-side ingredients. Can you explain a bit further what you're thinking of?
How will test-runner specific configuration be specified (or not) per ingredient/pantry?
I guess they would put it in whatever config file we choose in #17. The Roux test runner should also pass through command line arguments.
Is test execution time bound by a timeout or is it open-ended?
The Roux test runner should probably have some sort of configurable timeout.
A test entry point MAY assume that a Roux server is running and WILL receive the root URL for its ingredient as an environment variable named ROUX_INGREDIENT_URL.
Thinking about this further, I'm coming around to @lzilioli's view that it probably doesn't make sense to assume the server is always running. I'll change this to say that the test entry point MAY receive the ingredient URL via that environment variable.
I'm going to draft a new version of this RFC and submit it as a PR into a new doc/rfc folder. That way we can track the changes.