In the lifecycle of your Squish tests, there may be situations where you wonder why your test is taking more time than you would expect. You might also wonder why a well-running suite of tests is taking a significantly higher amount of time when running on a new version of your application.
The Test Results give a high level overview over the execution times. Application Lifecycle Management (ALM) tools even allow doing statistics on those times or are able to show trends. But what if we want to find out why a test run is not as fast as expected, or why it got slower at some point.
With Python comes a wide standard library, among it a profiler called cProfile. The general idea of a profiler like cProfiler is to have it measure how many times each function was called and how much time was spent in each function.
Getting Started with cProfile
In its simplest form, it would be just the following two script lines that have to be added to an existing test case script.
import cProfile
cProfile.run('main()')
run() is a convenience method that starts the measurement, invokes the given code and prints the results to stdout at the end. When using the Squish IDE, the output would show up in the ‘Runner/Server Log’ tab.
The drawback is that Squish also makes sure to execute the main() function by itself; so we would end up with two runs of main(), thus two runs of the test case. This can be fixed by renaming the main() function and creating a new one, that invokes the renamed version and takes care of the
profiling.
import cProfile
def orig_main():
pass
def main():
cProfile.run("orig_main()")
Having to change too much of the test script to get the profiling added (and removed), is cumbersome and error-prone. This becomes even more true when we want to take a bit more influence on the display of the profiling results, which requires some more lines of Python code to be added.
Using Python Decorators to Make Turning the Profiler On and Off Easier
To avoid having to rename functions and adjust existing lines of code each time, we can make use of Python decorators. They allow us to “decorate” an existing function by inserting a single line before the function definition. During execution, the decorator implementation is called instead of the “decorated” function. The responsibility of calling the function is handed over to the decorator implementation. We can use this to implement the profiling in such a decorator. This might sound a bit dry, so here is how it works in practice:
import cProfile
def profiled(functionToProfile):
def profileWrap(*args, **kwargs):
# Since we get passed a pointer to the function and not just a name,
# we have to give up the convenience function and do the steps (start
# measurement, run code, print results) ourselves now,
benchmark = cProfile.Profile(builtins=False)
origReturnValue = benchmark.runcall(functionToProfile, *args, **kwargs, "cumulative")
benchmark.print_stats("cumulative")
return origReturnValue
return profileWrap
So we created a decorator profiled, that we can now use to “decorate” our functions with. Whenever the function is called, even if it’s not in our hands like with main() in Squish test scripts, the code in the inner function is run.
We can store that in a dedicated file in, for example, the Test Suites Resources, so that import statements will pick up the file whenever we need it.
When we want to enable profiling of a particular function we just “decorate” it like this:
from squprof import profiled
@profiled
def main():
pass
Et voilà, when the test case is executed, it gets profiled and we see the profiler’s report in the output of the squishrunner.
Analysing the Results
When we make use of the above profiled decorator and apply it to an existing test case — one where we always wondered why it takes a bit slow — a report we receive could look like this:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.005 0.005 285.322 285.322 test.py:71(main)
1 0.008 0.008 1.710 1.710 test.py:22(loadAddressBook)
20 0.007 0.000 161.595 8.080 test.py:18(verifyEntriesExist)
20 0.021 0.001 26.050 1.303 test.py:32(addEntry)
20 0.081 0.004 77.531 3.877 test.py:41(editEntry)
20 0.306 0.015 17.445 0.872 test.py:54(removeEntry)
1 0.000 0.000 0.337 0.337 test.py:65(quitAddressBook)
220 76.312 0.347 161.309 0.733 test.py:7(searchByEmail)
82720 41.305 0.000 41.305 0.000 {object.children}
55484 45.119 0.001 45.119 0.001 {squish.waitForObject}
165 0.728 0.004 0.728 0.004 {squish.waitForObjectItem}
1 0.649 0.649 0.649 0.649 {squish.startApplication}
80 0.357 0.004 0.357 0.004 {squish.waitForObjectExists}
220 0.279 0.001 0.279 0.001 {test.verify}
121 0.068 0.001 0.068 0.001 {test.compare}
27720 0.014 0.000 0.014 0.000 {len}
1 0.000 0.000 0.000 0.000 {range}
388 120.063 0.309 120.063 0.309 {built-in method global}
A couple of things stand out and justify further inspection.
- A lot of time (161 seconds i.e., 2/3 of the overall runtime) is spent inside
verifyEntriesExist()/searchByEmail(). - A five-figure number of calls of object.children() and waitForObject() and the Python built in len(). Even though a single call seems to be speedy (percall: 0.000 and 0.001 numbers), it sums to a significant amount.
- The 76 seconds searchByEmail() is spending on stuff. The second column of the report (tottime) reports the time a function was spending doing stuff itself, i.e. it does not include the time it was just waiting on functions it called to return, contrary to the cumtime column, which does include that time. So of the 161 seconds spent within searchByEmail(), half was spent on calling object.children() and waitForObject() a lot, but the other half it spent on language level things, like looping, calculations and calling functions.
Note: One should not neglect the performance impact profiling can have on the test execution. A big part of that 76 seconds tottime of searchByEmail() comes from the overhead of around 100k function calls in combination with the profiling.
By the way, the “built-in method global” line accumulates the time that is actually spent interacting with the AUT, calls to clickButton(), mouseClick(), etc.
Tackling the Potential Issues
We would start with looking at the function implementations of verifyEntriesExist() and in turn searchByEmail(). We would watch for obvious things, and maybe we discover this badly optimized loop right away:
while cnt < len(object.children(waitForObject({"type": "javax.swing.JTable"}))):
Here, at least the waitForObject() call could be done once before the loop, saving a lot of repeated waitForObject() calls that now happen on each iteration, yielding the same object every time.
And this sample also shows a second class of issues: Usage of inefficient API. In that case, the len(object.children(table)) requires a lot more work than if we would have just used the rowcount property of the table.
When there isn’t anything obvious, we could repeat the profiling on a particular function, by decorating just that function, and rerunning the test.
@profiled def searchByEmail(email):
There is also always the possibility, that the effort needed for a performance improvement is not worth it.
What About BDD Test Cases?
The same works for BDD tests. It’s also possible to “decorate” step definitions, just make sure to have it below the existing decorator, so it is “invoked” first.
@Given("some precondition is met")
@profiled
def step(context):
pass
Diving in Deeper
- For influencing the output of the report you could make use of the pstats module. For example, you could ask it for a sorted report and place it into the Test Result using test.log().
- pyinstrument is a Python profiler that provides a better breakdown of the pieces of the profiling results, but is a 3rd party library only, that is not part of the Python installation shipped with Squish.
The post Collecting and Analyzing Execution Times of Squish Tests Written in Python appeared first on froglogic.