Do you have tests that fail randomly? Or maybe just periodically?
Categorizing Test Failures
Identifying and separating test failures related to defects versus flaky tests isn’t always as straight forward as it may sound.
Why important?
Time. Exponentially impacted time.
When test failures are not handled intelligently, manually troubleshooting and distinguishing the two can lead to a time engulfing maintenance nightmare.
What now?
Re-run failed tests multiple times. Either using your automated batch, CI process or directly from your scripts, logging then in the results the failure occurrence rate.
- Tests which fail should be run at least three times (or select an alternate threshold, with the goal of increasing this number over time; for example to 1/5 or 1/10).
- If the test passes just one of the three times, then it should be flagged to be further investigated for improvement, keeping in mind, it may indeed be the AUT, and not the test.
- If a test fails greater than a third of the time, then investigate further as a software bug or defect.
Keep in mind…
tests are most effective when produced with a specific goal in mind, not a series of items to validate along the way. Having longer tests with various dependencies creates a more difficult to maintain framework. This also lengthens the time investment required to identify the source of each issue – exactly where, what went wrong. As your test framework grows, the maintenance investment grows.
Think in terms of a Behavior Driven Test:
- Given x
- When y
- Then z
Too many Given’s and When’s can muddy the waters – making it more time consuming to pin point the root cause of the issue.
Start small, start smart, refine… then expand!
Remember; we’re always adding to our knowledgebase and other online resources to provide you with the most current and helpful information!
