Locating Bugs by Comparing Coverage of Two Tests

The typical answer to the question: Which source code line is responsible for a bug? is Use a debugger!

However, it’s not always that simple. Here’s why:

• Without good working knowledge of the code, it can be difficult to set a breakpoint at a pertinent location, enabling you to even start the investigation.
• Many problems are difficult to reproduce because the environment is unavailable or the issue doesn’t occur systematically.
• Issues are not always discovered by developers themselves, but instead reported by an integration team using a detail-lacking report – often requiring further interpretation.

The number of source code lines containing error candidates can be reduced using Squish Coco’s coverage information: simply compare the execution of two similar tests and analyze the difference between the passed and the failed test.

Here’s a simple example:

float inv2( int x ) 
{ 
    float r = 0.0; 
    if ( x != 0 ) 
        r = 1 * x ; // <- the bug 

    return r; 
}

Given the following test suite:

Test inv2( 2 ) fails because we expect a return value of 0.2, whereas this function returns 2.0.

In the CoverageBrowser, activate Execution Analysis mode by selecting Execution Comparison Analysis from the Tools menu. In the Executions list, check inv2( 0 ) as the reference and inv2( 2 ) as the test to analyze:

The Executions window’s coverage column indicates one source code instrumentation was executed more in the inv2( 2 ) test than was in the inv2( 0 ) test. A quick look at the Source window helps identify the line:

The expression r = 1 * x is wrong because multiplication is used instead of a division.

This technique of course has some restrictions:

1. It works only if the failed test execute more code than the passed reference test. This is the case if an error message displays in the failed test, or if when executing the failed test, the behavior is entirely different from the prior version.
2. It is necessary to identify tests which are similar to the failed test – which is not always easy.

To handle these two restrictions, Squish Coco has a dedicated algorithm coming in version 3.4.