Repository-wide testing of blueprints

[ksatirli]

I'd love to have the ability to run all tests of a repository, which might include multiple, dependent blueprints.

Here's some background:

• base defines a network stanza and some tools
• consul defines a Consul setup and generates TLS

Right now, shipyard test ./base works, because all dependencies are there - this is as expected.

On the other hand, shipyard test ./consul does not work, from the second scenario on, because at that point, shipyard destroy was run and that wiped out the resources (specifically network) that I created using the base blueprint.

I understand that scenarios should be encapsulated and welcome the approach of create, test, destroy but do think it needs support for dependent resources.

In Shipyard, this is fixed by depends_on and will block running of a blueprint if the dependencies aren't there. I think a method like this for tests to bubble up their requirements would be great to have.

If we think of testing blueprints in a single repository as a cohesive unit, I think it would make sense to be able to run shipyard test on the top-level. This then parses any .feature files it finds in ./test (or ./tests, which seems more correct as there should be multiple tests).

Those tests are executed, using the blueprints associated (as defined ... possibly in the feature file?) like so:

./test/consul has dependent blueprints defined as ./base and ./consul

• both of those are executed prior to running a test
  • a failure to run any of the child items (network, tools container, exec_remote etc.) results in tests not being run

When all is running, the test runner executes each scenario, cleaning up and re-creating the steps in the previously listed blueprints.

I realize this will make test execution a lot longer, but at the same time allow for testing the complete repository, which I think makes sense, as these are related resources.