Multi-User, Blackbox Testing with Squish Coco

Software development success is often achieved in a team of engineers, and part of this success is achieved through thorough, comprehensive testing. In some settings, source code is shared among all involved on a given team, but in many cases, the source code is secured, and only parts of it are available to a given engineer. Imagine the case where you have a distributed team of test engineers, all of whom do not have access to the application’s source code. Ideally, we would like to collate the manual tests of each test engineer and the unit tests of the developers into one global coverage report, to analyze the testing efforts of the whole team. In most code coverage tools, this sort of blackbox testing is not possible: without the source code, the tool is not able to produce coverage metrics. This is not only possible in Squish Coco, but easy and quick to achieve. In this week’s tip, we’ll cover how to handle multi-user testing in Squish Coco in order to get the coverage of all tests from all engineers, even if the source code is not available to all on the team.  

To demonstrate this, we will be using a C++ parsing program that acts simplistically as a calculator for standard expressions. This program is packaged with all versions of Squish Coco. In this example, we have a master user who has access to the source code, and additional test engineers who do not have access to the source code, but are completing manual tests. The developer in charge of the source code has written a number of unit tests.

Unit Tests

To begin, the developer executes the unit tests:

$ make clean
$ ./instrumented make tests

We can open the coverage report within the CoverageBrowser:

$ coveragebrowser -m unittests.csmes -e unittests.csexe

In the condition coverage view, we can see the coverage is at 45.581%. Taking a slightly deeper look into one of the functions, factorial, we see a 0.000% condition coverage. It’s clear that no unit tests have been written for the factorial function. 

Distributing the Application

The developer will now ship the instrumented application to the manual testers. First, he or she will instrument the application, so that the coverage record of the manual tests will be recorded.

$ ./instrumented make

This generates an executable for the parser program, called parser. The developer will ship this to the testing team for further testing. Whenever this program is run, and manual tests are entered into the parser, a *.csexe file will be generated. This is the execution report. 

Creating a Blackbox Database

If the intent is to keep the source code secure, the developer can create a ‘blackbox’ database in which execution reports can be imported, but no source code is shown. This is achieved through the following:

$ cmmerge -o parser_blackbox.csmes --blackbox parser.csmes

Manual Tests

With the executable distributed to the testing team, the first test engineer issues some tests to cover the factorial function:

> 3!

	Ans = 6

> 0!

	Ans = 1

> 1.54!

       Error: Integer value expected in function factorial (col -1)

And a parser.csexe file is now generated. 

This can be opened in the CoverageBrowser:

$ coveragebrowser -m parser_blackbox.csmes -e parser.csexe      

…where “parser.csexe” is the execution report generated by doing the manual tests. 

We see that, by using a blackbox database, the source code is not viewable, and only a  reduced amount of information is shown (i.e., the global coverage, a window for comments, etc.) Click Save before exiting the CoverageBrowser. 

Merging the Results

Once the manual testers have completed their work, it is up to the developer to merge all coverage data in order to get a full scope of the coverage. 

The developer will first need to import his or her unit tests. In other words, load the execution report:

$ cmcsexeimport -m unittests.csmes --title=‘UNIT-TESTS’ unittests.csexe

Finally, the developer can merge all reports into one, called “all_tests.csmes.”

$ cmmerge -o all_tests.csmes -i parser.csmes parser_blackbox.csmes unittests.csmes

Opening this in the CoverageBrowser, we see the following:

Note that in the bottom right hand corner of the window, we can toggle the execution view to show coverage levels from manual tests and unit tests (located in CppUnit). 

In the above screenshot,  we also verify that factorial is now 100% covered, owing to our manual tests. 

Generalizing to Multiple Testers

The example above covered only one developer-tester pair, but the same steps can be generalized to multiple testers. 

To collate all results, issue the following:

$ cmmerge -o all_tests.csmes -i parser.csmes uniittests.csmes parser_blackbox.csmes parser_blackboxUSER2.csmes ... parser_blackboxUSERN.csmes

Note that each user should have different names for their tests. In the above screenshot, we’ve named the one tester’s efforts “TESTER1.” 

The post Multi-User, Blackbox Testing with Squish Coco appeared first on froglogic.