Failed scenario reported as success if followed by successful scenario
Description
It seems that the jbehave run will pass if there is a scenario that fails that is followed by a scenario that passes. I have created a minimal project to show this:
Building this with maven will give a green build, but the first scenario fails, so build should fail. Same thing if the story is run from within intelliJ.
This can be recreated from version 4.0+. The exact same code will fail (as expected) using version 3.9.5.
Debugging the code I see that the state is reset between the scenarios but the failures are not added to the context before this.
I have no idea, I agree with you, IMHO it should not hide any failures by default. I let the JBehave team answer this question.
Jonas Häll August 26, 2020 at 12:05 PM
Thanks, that does the trick, but perhaps the MostUsefulConfiguration is not so useful if it will hide failures in all scenarios but the last one? Anyone who knows the reason for the change in behaviour?
Nicolas Filotto August 26, 2020 at 11:25 AM
Edited
Maybe you could set resetStateBeforeScenario to false
Your configuration would then be:
public Configuration configuration() { MostUsefulConfiguration configuration = new MostUsefulConfiguration(); configuration.useStoryControls(new StoryControls().doResetStateBeforeScenario(false)) .storyReporterBuilder() .withFormats(Format.CONSOLE) .withRelativeDirectory(""); return configuration; }
It seems that the jbehave run will pass if there is a scenario that fails that is followed by a scenario that passes. I have created a minimal project to show this:
https://github.com/JonasHallFnx/jbehave-bug
Building this with maven will give a green build, but the first scenario fails, so build should fail. Same thing if the story is run from within intelliJ.
This can be recreated from version 4.0+. The exact same code will fail (as expected) using version 3.9.5.
Debugging the code I see that the state is reset between the scenarios but the failures are not added to the context before this.