Add @BeforeAll and @AfterAll hooks
aslakhellesoy opened this issue · 128 comments
People keep asking for BeforeAll and AfterAll hooks. I haven't had a big need for them myself, but it seems to me this would be sufficient:
public class GlobalHooks {
private static boolean dunit = false;
@Before
public void beforeAll() {
if(!dunit) {
Runtime.getRuntime().addShutdownHook(afterAllThread);
// do the beforeAll stuff...
dunit = true;
}
}
}If this doesn't cut it for you, please explain in this ticket, and maybe we'll add special support for it.
Does that fire when the process dies, or just the current cucumber run? What if you wanted to do multiple runs without killing the process? (In an Aruba 5.2 stylee)
It wouldn't be triggered by a VM shutdown. It would be part of the runner's life cycle and fire after the last scenario.
For Java I think they would have to be defined as static methods as well.
What would happen if you try to embed data into the report in this
"BeforeAll" hook? It seems your example would work well for running some
code only once at the start of the scenarios, but any embedded would be
inserted into the first scenario that runs that code and not at the root
level of the report.
Furthermore, I supposed embed wouldn't work in the shutdown hook either in
the above case. Given scenario.embed is used for embedding data in
reports, I'm guessing there would have to be some new object reference in
order to embed data in the reports for @BeforeAll/@afterall hooks outside
of the scenarios.
Cheers,
-Tim
On Thu, May 2, 2013 at 6:46 AM, Aslak Hellesøy notifications@github.comwrote:
It wouldn't be triggered by a VM shutdown. It would be part of the
runner's life cycle and fire after the last scenario.For Java I think they would have to be defined as static methods as well.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/515#issuecomment-17333288
.
What would happen if you try to embed data into the report in this "BeforeAll" hook?
From a BeforeAll/AfterAll hook you would have optional access to a Report object with similar embed methods as the Scenario object available to Before/After hooks. Embeddings embedded from a BeforeAll would end up at the top of the report, before the first scenario. Similar for embeddings embedded from AfterAll - they would end up in the report after the last scenario.
If you attempt to do a Scenario.embed from an AfterAll hook you would get an exception - the scenario would be "closed" by then.
We definitely have many uses for this. Currently we use a static
initialiser (which I think we got from this forum) to start up embedded
databases, Hadoop's Mini MR cluster among other things.
It would be good to have a clear annotation such as @BeforeAll and
@afterall rather than a specific method name with an @before tag.
On 2 May 2013 07:37, Aslak Hellesøy notifications@github.com wrote:
What would happen if you try to embed data into the report in this
"BeforeAll" hook?From a BeforeAll/AfterAll hook you would have optional access to a Reportobject with similar embed methods as the
Scenario object available to Before/After hooks. Embeddings embedded from
a BeforeAll would end up at the top of the report, before the first
scenario. Similar for embeddings embedded from AfterAll - they would end
up in the report after the last scenario.If you attempt to do a Scenario.embed from an AfterAll hook you would get
an exception - the scenario would be "closed" by then.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/515#issuecomment-17341593
.
+1. I want having multiple BeforeAll and AfterAll supported (opposite to not like World)
Any movement on this issue? I'd love to have it. Of course, I'm just one person. ;-)
Those of you who need it - how about sending a pull request?
+1 because the trick above won't work for AfterAll
@quantoid It's not a trick, it's Java! Why do you say that it does not work for AfterAll? The only Issue I see is with reports.
The @AfterAll hook would be the afterAllThread variable in Aslak's code.
Added this snippet to the Java Hello World Example:
@Before
public void beforeAll() {
if(!dunit) {
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
System.out.println("snake!");
}
});
System.out.println("badger");
dunit = true;
}
}After changing the runner to use the progress formatter, when I run mvn test, I get the desired behaviour:
badger
.........
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.466 sec
snake!
Sorry, I can see how it works in Java, I should've added "...in Groovy" but could be wrong about that too.
I'd want both 'all' hooks to be able to access variables that other glue can access, i.e. those in the Script scope.
Same thing in the groovy-calculator example with BeforeAll and AfterAll, accessing the World object that contains shared variables.
class CustomWorld {
def value = null
def dunit = false
// ...
}
Before() {
if (!dunit) {
dunit = true
println "badger"
value = "snake!" // sets the value in the World
addShutdownHook {
println value // reads the value from the World
}
}
}Note that if you change the value variable in other steps, the shutdown hook is not going to print snake!, so the value is not determined when the closure is defined but when it is called (at the very end).
This change to a step will make the AfterAll closure print mushroom:
Then(~"the stored result should be (.*)") { double expected ->
value = "mushroom"
assert expected == result
}Of course it works even if you define the value variable outside of the World object.
Awesome, thanks! Where does the addShutdownHook come from? There seem to be so many classes mixed in to the scope of these closures.
@quantoid No magic there, it's part of the Groovy Object API
Hmm, IMHO would be better to tear stuff down as part of the feature execution block rather than when the JVM shuts down.
Hi,
I'd like to create a test database then destroy it after all the tests are done. The static boolean works for the @before tag, creating my database only once.
But how would I drop the database only after all my tests are done?
@TarekSaid this is a bug tracker, not a support forum. Please ask on the cukes google group. See https://github.com/cucumber/cucumber-jvm/blob/master/CONTRIBUTING.md
This will work OK for as completely global before/after, but it doesn't work if you want the behavior per feature file. For example, I may have 5 different feature files, each one having 10 different scenarios. Per feature, I may want a before/after. This workaround isn't per feature, but is per EVERYTHING. A per feature before/after would be nice.
For example, JBehave has a @Before/AfterStories. The code discussed in this thread would be equivalent to that, so this annotation isn't really necessary. It also has @Before/AfterScenario, which is equivalent to Cucumber's @Before/After. However, JBehave also has a @Before/AfterStory (singular) which isn't really doable with Cucumber-JVM (that I can tell). This would be very useful.
If you think about it, JUnit has it, too, with @Before/AfterClass.
Maybe I'm blind and there is a way to do it in Cucumber... but I can't see it.
@tschmal before/after feature hooks are out of the scope of this issue. They were discussed several times on The Cukes Google Group, so you might search there. In a recent thread I explained how to deal with feature tags. If you still think you need it, please write on the list first and explain your problem.
You don't need to do this in cucumber. Use the @BeforeClass and @afterclass annotation from within the JUnit test used to run the cucumber tests. This has the benefit of running only for the features specified by the paths or tags options.
@RunWith(Cucumber.class)
@Cucumber.Options(format = {"html:target/cucumber-html-report", "json-pretty:target/cucumber-json-report.json"})
public class RunCukesTest {
@BeforeClass
public static void setup() {
System.out.println("Ran the before");
}
@AfterClass
public static void teardown() {
System.out.println("Ran the after");
}
}@wjpowell there is nothing in the Cucumber runner that handles @BeforeClass/@afterclass. Have you actually tried this?
@aslakhellesoy I've tried it, and it works. The Cucumber runner calls super.run which handles the @BeforeClass and @afterclass annotated methods, as well as any @ClassRule defined.
Thanks for the clarification. I thought it might be something like that.
Pull request created for this issue: #672
Just waiting for someone to take a look and approve it if its all good.
Would love to see this included in the latest version :)
I just used the BeforeClass and AfterClass annotations today and they worked perfectly. I'm not sure why we need to add our own BeforeAll / AfterAll hooks.
@mattwynne, I see a few reasons:
- @BeforeClass and @afterclass are not part of the cucumber and having them in the runner class as static has no benefit but has a few downsides (usually a DI container is used and some components are injected in stepdefs classes to be used for "setUp/initialization" - how do you get those in the runner class on static context?? plus polluting the runner class with test logic)
- when you want to select a feature in Intellij Idea and run it, the cli.Main class will be used and your methods from runner class with @BeforeClass and @afterclass will never run.
- I don't know how many of you had this problem but in the last two years I used cucumber a lot, and I can say that because of small things like the setUp/tearDown for cucumber tests, your code for tests will get messy and hard to maintain because of the static state. For an example cucumber test, yes you can use in the runner @BeforeClass and @afterclass just to say that works, but in real world projects is not the same, you'll encounter all kinds of issues and something like @BeforeAll and @afterall helps a lot. That's my opinion and I might be wrong, but that is way I created this PR.
@mattwynne Because those annotations are from JUnit and they only work when you use the JUnit Cucumber runner.
Many people use Cucumber-JVM with the command-line runner, without JUnit at all, and then this would not work.
Good points!
I'm reopening this ticket since I realised that #678 will not work at all when there is DI involved. More details in that issue.
Is anyone working on this at the moment?
It this feature in the roadmap in a near future?
I have found the need of this feature several times in different projects.
And neither addShutdownHook nor @afterclass covered my needs, since we execute tests from sbt, addShutdownHook is only executed on JVM shutdown, while this does not happen until sbt is explicitly closed and I don't want to executed our tests in a forked JVM.
+1 for this feature. Is anyone working on it?
👍 here as well
+1
No word for over a year though, so I'm not holding my breath...
Pull requests accepted, but they need to cooperate with Dependency Injection.
Additionally it might be a feature that comes in when Gherkin 3 gets integrated. Glad to know people want it, but as with any OpenSource project, it moves at the speed of the volunteers that work on it.
Hi All,
I am having 5 Cucumber JUnit runner classes for parallel test execution, I want to concatenate all the JSON outputs together. Is there any Cucumber annotation that can work after all the parallel executions or can anyone suggest me other options which suits my requirement.
Thanks in advance,
Anite S
@AniteSridhar This is not a support forum. See: https://cucumber.io/support
+1
Junit BeforeClass and AfterClass annotations work fine if you are using a JUnit runner. No issues with the static state.. Works fine with Page object Model for me.
I don't see a resolution for this. People who have mentioned to use @afterclass don't understand what we're trying to solve here. We want a method that executes after the entire cucumber test run. The @afterclass doesn't work for this. It throws errors. If anyone has a solution it would be awesome if they take the time to share. thanks. -AD
@andresd24 @afterclass executes after the entire cucumber test run if you are using a Junit Runner. If you insert your feature tag into the runner, Junit will run all the scenarios in that feature and when it ends it executes the AfterClass. The irony is that, If you try running directly from your feature file Junit will not run..... The only way to run this is to run directly from your Junit Runner via the tags... You can even extend this junit runner class with abstract TestNG cucumber Test Class and also include TestNG @BeforeClass and @afterclass this will run before and after the tags when you specify the class or suite in your testng.xml file.
@RunWith(Cucumber.class)
@CucumberOptions (
monochrome = true,
features = "src/main/resources/features/Newtest",
plugin = {"pretty", "html:target/cucumber-html-report"},
tags = "@editNewtest"
)`
public class CukesRunner extends AbstractTestNGCucumberTests {
@org.junit.BeforeClass
public static void junitsetup() {
loadandlogin();
}
@org.junit.AfterClass
public static void junitteardown(){
browser.closeWebBrowser();
}
@org.testng.annotations.BeforeClass
public static void testNGsetup() {
loadandlogin();
}
@@org.testng.annotations.AfterClass
public static void testNGteardown(){
browser.closeWebBrowser();
}
}
`
Hi @OlufemiAdeOlusile. The @org.junit.AfterClass just doesn't want to run for me. We solved this by extending the class to an abstract test class that adds a addShutdownHook to the Runtime in the @BeforeClass that generates the reports....
TestClass.java:
@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty","html:target/cucumber"})
public class IridiumDemoTest extends AbstractTest {
}
AbstractTest.java:
public abstract class AbstractTest {
@BeforeClass
public static void setUp(){
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
try {
//generate report;
} catch (Exception e) {
e.printStackTrace();
}
}
});
}
}
Thanks for sharing TestNG since we'll look into using that in the future. Hopefully we can get it to work the way you mention.
@andresd24 your welcome, I don't think the problem is with the Junit Afterclass.. Rather look at your implementation again and find out if the afterclass gets the same instance after your tests run. Am sure this will be fine.. hehe..
No, the workaround is not sufficient. This is because it only cleans up test state when the VM terminates, and not before continuing to execute other (JUnit) tests. Lets see if we can fix the problems with #678 .
+1 for this issue
+1 as well
This is super useful for testing an application, and ensuring that the app is closed out completely at the end of a test run.
+1 for this issue
+1 for this issue
+1 for this issue
+1 for me here as well. I am going to try and make the problem go away with other means, but it would be really nice if Cucumber naturally supported this or made it otherwise more intuitive as to how to use it. I'm going to explain my issue in case that helps at all.
Currently using "Background" to load data into 3 different data stores in a Vagrant VM, then setting up Scenario Outlines for the purpose of exercising API endpoints with a variety of conditions, the API reading from the 3 data stores. It works, but as soon as I have the Background add multiple data sets (which are necessarily several thousand rows in some cases; not several thousand defined in the feature file, but by nature of the data being programmatically generated for Cassandra), the time to execute just blows up. The issue for me is that I don't necessarily want the same data set loaded for every feature file, which means in order to do Before/AfterClass, I'm going to have to get creative. Additionally, with some approaches I could lose the handy mapping of Cucumber->POJOs that align with my data structures, plus the readability of "what is the condition we're setting up before testing this set of endpoints." So I'm not able to just easily do one of the simple proposals here with Before/AfterClass.
Cucumber-Java has proven itself pretty popular at my company over the last year, but I'm basically forging the new territory by proving its usefulness, and this is the biggest application I've been tasked with sorting out. It's really nice to have something readable and clear like the Gherkin syntax, but this cumbersome lack of "set up the database scenario once before the scenario outline, not 20x for every example set of variables" is pretty awkward. I get that there are best practices and workarounds for most scenarios, but when you lose the clarity in the process...
JBehave has a sane implementation of hooks. Unfortunately the JUnit runner is not entirely bug-free against newer versions which cause some issues in builds and IDE info. But it could be used as a reference implementation, or we could instead, address the issues there.
@AfterAll
public void afterAllScenarios () {
driver.quit();
}Would be readable, my vote goes for such a hook being present.
+1
We need @BeforeAllor @AfterAll- because when we use Spring's @Autowired we are not able to use the @BeforeClass that is static
- 1 for the issue
+1 for this issue
I've been waiting for this forever. I can't believe it's still not supported.
It's 2016 already.
I've just come upon this issue, as I am tying to setup the load of some specific test data before a large series of scenarios for a specific feature, then I want to remove all data I have created for this feature at the end.
At the moment, as a work around, I have created before/after steps that do the work above and are called by relevant tagging;
e.g. '@load_test_data_for_feature_X' and '@remove_test_data_for_feature_X'.
These are applied to empty scenarios at the start and end of the file, e.g.
@load_test_data_for_feature_X
scenario: PREP - load test data for feature X
@remove_test_data_for_feature_X
scenario: CLEANUP - remove test data for feature X
this seems to work, but as with all other posts above, would really like to see @Before_Feature and @After_Feature hooks, as these would be infinitely more flexible than my clumsy workaround.
+1 because it's useful and reduces global code redundancy :-) Plus code reviews will be nicer if we don't have to explain that though the frameworks supports Before it doesn't support BeforeAll.
Additionally, when building a Cucumber instance that is triggered by code reviews, for example, being able to validate that all required systems have been correctly implemented and set up prior to execution is a godsend, and on the flip side, ensuring that same infrastructure has been correctly dismantled can prevent numerous issues on the build engineering side of a truly autonomous testing framework.
totally. 100% agree that's the proactive and only way to do it.... it's
been done for a while but only by visionary engineers.
On Mon, Oct 17, 2016 at 7:16 PM, Link notifications@github.com wrote:
Additionally, when building a Cucumber instance that is triggered by code
reviews, for example, being able to validate that all required systems have
been correctly implemented and set up prior to execution is a godsend, and
on the flip side, ensuring that same infrastructure has been correctly
dismantled can prevent numerous issues on the build engineering side of a
truly autonomous testing framework.—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#515 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APAWQhn-b79gUsr3gK9_vqAYAnpbMSADks5q1A_igaJpZM4An6aA
.
I'm grateful that @aslakhellesoy reverted the prior implementation because it would be a huge mistake not to support DI. My whole use case for needing these hooks is to use a class marked as singleton from the Injector for reporting purposes.
Unfortunately, I'm not yet fully versed in the whole concept of worlds and the internal implementation of the runner but hoping someone solves this in the near future.
+1 from me.
PS: I'm currently working around this limitation via a static injector. I'm assumming this won't encounter any implications...
public class DefaultInjectorSource implements InjectorSource {
public static Injector injector = Guice.createInjector(Stage.DEVELOPMENT, CucumberModules.SCENARIO, binder -> {
binder.bind(ExportConfig.class).to(PropertiesExportConfig.class).in(CucumberScopes.SCENARIO);
binder.bind(StepDefinitionExporter.class).in(Scopes.SINGLETON);
});
@Override
public Injector getInjector() {
return injector;
}
}public class LocalTests {
private static final Logger LOGGER = LoggerFactory.getLogger(LocalTests.class);
@BeforeClass
public static void boot() throws Exception {
LOGGER.info("boot!");
}
@AfterClass
public static void shutdown() throws Exception {
LOGGER.info("shutdown!");
DefaultInjectorSource.injector.getProvider(StepDefinitionExporter.class).get().doExport();
}
}+1
+1
Still hoping...
with multiple runners and multithreaded execution yet without an "afterAllThreads" to close the driver thread pool is inconvenient, to say the least ... need better multithreaded support
+1
+1
+1
+1
+1
+1
+1
++
+1
+1
+1
+1
+1
+1
+1
Is anybody of developers can comment this issue?
I've followed this thread with
compile 'info.cukes:cucumber-java:1.2.5'
compile 'info.cukes:cucumber-junit:1.2.5'
wrt comments
- I found execution happens like BeforeClass -> Before -> After -> AfterClass
But I found @ ClassRule does not get executed.
I've 2 methods in @ ClassRule
@OverRide
protected void starting(Description desc)
@OverRide
protected void finished(Description desc)
cc: @wjpowell
For those new to cucumber, if you've doubt wrt comment #515 (comment)
this is the answer https://stackoverflow.com/questions/16739347/alternate-way-to-run-cucumber-without-junit
I have found the easiest way to do this is to create a new cucumber "Backend" and then use buildWorld and disposeWorld methods to hook into.
Another example solution using JUnit with class rules.
@RunWith(Cucumber.class)
public class Some02IT {
@ClassRule
public static TestRule classRule = (base, description) -> new Statement() {
@Override
public void evaluate() throws Throwable {
try{
setupExpensiveFixtures();
base.evaluate();
} finally {
tearDownExpensiveFixtures();
}
}
};
}@marinat just create a class in the cucumber.runtime package (in your project) that implements the class "Backend" (from the same package). We implement our initialization in loadGlue and static cleanup (disposing of temporary AWS queues) in disposeWorld.
+1
+1
Still waiting.
+1
I realise this has been open for a long while. Thanks everyone for the +1 comments.
One way to increase the likelihood of this happening is to give financial support to the volunteers who maintain Cucumber: https://opencollective.com/cucumber
I thought I posted an answer to this? We used it to load all of our configuration, initialize log4j properly, and close down temporary SQS queues. Our test lead documents our e2e/cucumber stuff here: http://connect.cd/2017/10/introduction-to-microservice-test-automation-with-cucumber-jvm-and-openapi/
class must be in cumber.runtime package as that is all it scans.
package cucumber.runtime;
import cucumber.runtime.io.ResourceLoader;
import cucumber.runtime.snippets.FunctionNameGenerator;
import gherkin.formatter.model.Step;
import java.util.List;
public class BeforeAfterHooks implements Backend {
private Glue glue;
private List<String> gluePaths;
public BeforeAfterHooks(ResourceLoader resourceLoader) {
}
@Override
public void loadGlue(Glue glue, List<String> gluePaths) {
this.glue = glue;
this.gluePaths = gluePaths;
// do your before hooks here
}
@Override
public void setUnreportedStepExecutor(UnreportedStepExecutor executor) {
}
@Override
public void buildWorld() {
}
@Override
public void disposeWorld() {
// do your after hooks here
}
@Override
public String getSnippet(Step step, FunctionNameGenerator functionNameGenerator) {
return null;
}
}
I wanted to clenup some data after all scenarios from feature file. I believe @afterAll ("@tag") will be ultimate solution.
I tried with @After("@tag") but it gets executed after every scenario from feature file.
@mpkorstanje Do you think we can progress #678 this now? This was reverted as it was failing on DI.
I am not seeing a working pull request. And I have also not seen any fundamental discussion about how this is supposed to work with java8 and DI.
+1
it's really need for our testing. We use @BeforeClass @afterclass TestNG to prepare data on cloud service for group of tests. It's very expensive to create 'service project' for each scenario, thats why we want to use only one 'service project' for group of scenarios.
+1
+1
+1
+1
In addition to@BeforeAll, can we have @BeforeFeature as well, something similar to Specflow? These hooks really makes framework maintenance easy.
Its really surprising to see this issue open since 2013!!