Squish 6.3 introduced the image lookup feature. It allows capturing an image of an AUT control during test case recording. During test replay it allows finding the screen position of the specific image and interaction by clicking or tapping at the found position. It is the only way to automate custom-drawn components. In theory any component can be automated using image based approach. However, the simple image comparison implemented in Squish 6.3 allows only exact image matches. This causes a number of difficulties in practical applications:
-
- Large number of images needs to be added to the test suites in order to cover every component state of interest;
- Different sizes and rendering styles of particular components on different platforms require multiple images and special handling on the test-script side;
- Displays of different pixel density often require graphics of varying sizes;
- Small, indeterministic rendering differences make strict image search ineffective;
- Finding components in complicated renderings is often difficult.
While the image lookup does support transparent search images, the algorithm for handling transparency is geared towards usage with original images used by the AUT. That can lead to unexpected results when used to relax image lookups. In order to achieve predictable results we recommend using images with binary transparency only, i.e. images with areas of either full transparency or no transparency.
In the upcoming Squish 6.4 release the image lookup feature will be expanded with several relaxed matching modes. The new modes allow greater flexibility in image lookups and help to reduce the number of different images that need to be attached to the test suite. It also allows grouping multiple images and using such groups to preform image lookups.
Image groups
Image groups are sets of images that are equivalent representations of the particular element; i.e. they show different ways a component can look like. Any occurrence of any of the group’s members is considered a group’s occurrence too. Grouped images are a perfect way of representing controls that are rendered differently on different platforms or using different style settings.
Image Selection dialog displaying an image group
You can convert any existing lookup image into an image group by adding further images to it. This is especially useful if you need to adjust an already existing test suite to a new environment. Usually test script itself requires no changes in such cases.
Per-pixel tolerance
The per-pixel tolerance mode is best for handling slight discolorations of the image. Such differences can be the result of color differences between different platforms or application styles, anti-aliasing or rendering a semi-transparent overlay on top of it. The tolerance can be specified as maximal difference between pixel values on each of the RGB channels:
waitForImage( "open_icon", { match: { type: "pixel", tolerance: 10 } } );
Image cross-correlation
The normalized cross-correlation mode is best for handling localized differences between the search image and the image on the screen. It computes the normalized correlation between the search image and the desktop screenshot. Then, the correlation is compared against a configurable ‘threshold’ parameter
waitForImage( "green_rect", { match: { type: "normcorr", threshold: 0.98 } } );
You can use the “normcorr” method in order to locate images with aliasing differences on even partial occlusions.
Visually similar images can have large color differences in certain pixels.
Multi-scale image lookup
The same component can render its contents in variety of sizes. Different DPI settings and scaling modes or components scaling with window size are only a few of the possible reasons for such size changes. The multi-scale image lookup mode allows finding resized occurrences of a lookup images.
Toolbar of the same application rendered under different DPI and scaling modes.
You can use a single search image in order to find it’s occurrences in sizes that vary between half of the original size and double the original size using the following syntax
waitForImage( "open_icon", { match: { type: "multiscale", threshold: 0.97 } } );
Image lookup IDE support
The Squish IDE can help finding usable threshold values. While recording a test script, Squish IDE will display an ‘Image Lookup Preview’ dialog that allows experiments with different lookup modes and parameters.
The Image Lookup Preview dialog
The dialog presents in real time the results of image lookup process.
It is also possible to analyze the failed image lookup after the test completion. If you enable the screenshot logging on script errors
testSettings.logScreenshotOnError = true; mouseClick( waitForImage( "open_icon" ) );
then it is possible to open the image lookup preview dialog in order to find the best matcher parameters in the failed case. It is especially useful in case of unattended test executions or test runs on remote machines.
Image Lookup Preview on a test failure
The new features provide more flexibility and reduce the manual work during test creation and maintenance. Proper application of the new lookup modes minimizes the the amount of necessary image files and allow truly multi-platform testing.
The post Upcoming Squish GUI Tester 6.4 Feature: Fuzzy Image Lookup appeared first on froglogic.