Weird behaviour when stories are cancelled

Description

I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

Issues it can demonstrate:

Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report

This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

Scenarios: Total=1, Successful=0, Pending=0, Failed=1

The target/jbehave/view/reports.html shows just

No second scenario

JBehave may report cancelled scenarios as successful.

Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

Scenarios: Total=2, Successful=1, Pending=1, Failed=1

Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?

Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.

I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...

Assignee

Unassigned

Reporter

Dmitry Andrianov

Labels

None

Components

Affects versions

Priority

Major
Configure