Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me). Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.
Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like
Scenario: slow and failing test
Given a default state
When I do something slow (FAILED)
java.lang.InterruptedException: sleep interrupted
STORY CANCELLED (DURATION 2 s)
No second scenario
JBehave may report cancelled scenarios as successful.
Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like
Which seem to be just wrong given that target/jbehave/view/test.test.html shows:
Scenario: slow and failing test
Given a default state
When I do something slow
Then nothing should happen (FAILED)
java.lang.InterruptedException: sleep interrupted
Scenario: scenario description
STORY CANCELLED (DURATION 3 s)
So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.
I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present. Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.
Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue
Issues it can demonstrate:
Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like
Scenarios: Total=1, Successful=0, Pending=0, Failed=1
The target/jbehave/view/reports.html shows just
Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s)No second scenario
JBehave may report cancelled scenarios as successful.
Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like
Scenarios: Total=2, Successful=1, Pending=1, Failed=1
Which seem to be just wrong given that target/jbehave/view/test.test.html shows:
Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s)So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.
I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...