I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.
Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue
Issues it can demonstrate:
This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like
Scenarios: Total=1, Successful=0, Pending=0, Failed=1
The target/jbehave/view/reports.html shows just
No second scenario
Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like
Scenarios: Total=2, Successful=1, Pending=1, Failed=1
Which seem to be just wrong given that target/jbehave/view/test.test.html shows:
So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.
I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...