Creating Code Coverage Statistics with xdebug


By: Stefan Priebsch. Other Articles in the Series. Part One: Introducing xdebug. Part Two: Tracing PHP Applications with xdebug. Part Three: Profiling PHP Applications With xdebug. Part Four: Debugging PHP applications with xdebug. Part Five: Creating Code Coverage Statistics with xdebug. Welcome back to the fifth and last installment of our series of xdebug articles on Zend DevZone.



 
 

Related Links:

Original URL

http://devzone.zend.com/1156/creating-code-coverage-statistics-with-xdebug/

Content

Cached Text (at the time of saving)


By: Stefan Priebsch
Other Articles in the Series
Part One: Introducing xdebug
Part Two: Tracing PHP Applications with xdebug
Part Three: Profiling PHP Applications With xdebug
Part Four: Debugging PHP applications with xdebug
Part Five: Creating Code Coverage Statistics with xdebug
Welcome back to the fifth and last installment of our series of xdebug articles on Zend DevZone.
xdebug is the swiss army knife extension for PHP developers. In addition to some nice features like beautifying var_dump() output, adding stack traces
to error messages, xdebug supports tracing, profiling, and debugging, which were covered in the previous articles.
Today, we will have a look at another great feature of xdebug – creating code coverage statistics. Code coverage
statistics show how many times each line of the code has been executed. Conversely, they also show
which lines of code have not been executed, which is in fact much more interesting.
As we know, each program has different execution
paths (unless a program has no branches at all). A program’s execution path depends on the input data, which includes the parameters passed to the program, but any data in the database or from other data sources.
As an example imagine a website
that allows you to create a user account. This account must be verified by visiting a link that is emailed to you.
If you log in to a website, you might see a different response depending on wether you have already confirmed your account or not, though the GET or POST data passed to the script is the same. In this case, the program’s execution path depends on the confirmed flag of the user record in the database.
Code coverage statistics of one single program run are not very useful.
But when we collect and aggregate code coverage statistics of many program runs (with different execution paths), the coverage statistics show us which program parts have not been executed. These parts are either obsolete or should be tested.
The preferred way of creating code coverage statistics is to run unit tests while collecting coverage statistics of the
tested code. Note that coverage statistics are not created for the unit tests themselves, but of the code that we are testing. A unit test is a test of a class, module, or any other part of an application. Unit tests make it much easier to feed different input data to the code to test than if we would be testing an application in whole.
Technically, creating code coverage statistics is a rather complex issue that involves various tools.
Coverage statistics in xdebug are basically an array for each file where the line number is the array key and the array value is
the count how often this line has been executed. We’ll need some other tools to visualize these
coverage statistics.
To run automated tests, and collect the coverage statistics that xdebug provides, we will use PHPUnit.
To make testing with PHPUnit easier, we will use the build automation tool phing,
which allows us to define a file set of all tests, run PHPUnit tests on each of these files, and gather
and aggregate the collected coverage statistics. phing also converts the coverage statistics to a nicely formatted
HTML report using XSL.
Assuming that you (still) have xdebug installed on your system, we’ll need to install the other
tools before we can start.
Installing phing and PHPUnit
phing is based on the build tool Apache Ant, which is written in Java. phing was created so that
no Java Runtime Environment would be required to automate build processes on PHP development systems.
The main advantage of phing over Ant is that you can extend phing by writing your own PHP classes.
To extend Ant, you need to write Java code.
To install phing, you need PHP 5 with XML and XSLT support. On Unix, you must compile PHP with the
configuration switch --with-libxml. On Windows, XML support should be enabled by default.
To make sure you have XML support enabled, check the output of phpinfo() for a libxml
section.
To activate XSL support in PHP on Windows, you must load an extension by adding
extension=php_xsl.dll
to php.ini. Remember to restart your web server to make the change effective.
On Unix, the XSL extension is named php_xsl.so and will only be present, if you have
configured PHP using the switch --with-xsl. When the XSL extension has been loaded in PHP,
a section XSL will appear in the phpinfo() output.
phing is deployed as as PEAR package. To install phing, you must have a PEAR environment on your system.
Run pear version at the command line if you are not sure wether PEAR is installed.
As I write this article, the current PEAR version is 1.6.2.
If you have an older version, you should upgrade PEAR first:
pear upgrade-all PEAR pear channel-update PEAR
Normally, when we install phing with the switch --alldeps, all dependencies like PHPUnit
should be installed automatically. PHPUnit, however, has moved from PEAR to its own channel server, and
the classes were renamed from PHPUnit2 to PHPUnit (instead of PHPUnit3, as PEAR would have dictated).
Therefore, PEAR currently fails to install PHPUnit 3.x from the channel pear.phpunit.de. The best
idea is to install the latest version of PHPUnit3 manually before installing phing:
pear channel-discover pear.phpunit.de pear install pear.phpunit.de/phpunit
Now we can install phing.
As I write this article, the current version of phing is 2.3beta1. The PEAR installer
will automatically install the latest version of phing, but since this version is in beta state,
you should add the command line swich --force, otherwise an older version of phing might
be installed instead.
pear channel-discover pear.phing.info pear install --force --alldeps pear.phing.info/phing
phing is based on XML files that define the various targets (“jobs”) of a build process, like
build, test, or – for web sites – deploy which might copy the web site
right to the live server. The XML format makes phing builds portable across different platforms,
unless, of course, you rely on
Creating unit tests with PHPUnit
Now that we have installed all necessary tools, we must create some unit tests.
I would recommend keeping the tests in the same directory tree where your production code
is kept. When deploying your application, you can always leave out the test files, unless you
want to create version that allows your end users to run the unit tests as well.
assertEquals(0, $foo->getNumberOfBars(), 'Foo is not empty'); $bar = new Bar; $foo->addBar($bar); $this->assertEquals(1, $foo->getNumberOfBars(), 'Foo has no bar'); } } ?>
This example shows a rather simple unit test that makes not much sense by itself in the real world, but
serves to illustrate the purpose. A real-world test case would probably contain more
test methods and does not even use a test fixture.
We assume that autoloading is used, so that the Foo and Bar classes are loaded when
we are trying to instantiate the objects.
Creating a phing build file
For the purpose of this article, our build file will have only one task that creates code coverage statistics.
By default, phing build files should be named build.xml. The following listing shows the phing build file that creates code coverage statistics:

Like in PHP itself, always use forward slashes as directory separator, even on Windows.
We are using two filesets, php and tests. The first fileset
contains all PHP files of our application. These are the files we want to create the code
coverage statistics for. The second fileset, tests, contains all unit tests of
our application. Please note that all unit test files are explicitly excluded from the first fileset.
Each fileset has a base directory, src that is specified in an attribute of the fileset tag.
You can include or exclude files using wildcards, with * matching files in the given directory,
and ** matching the subdirectories.
In the code_coverage build target, the coverage-setup task sets up a
database with code coverage data provided by xdebug via PHPUnit. This database is in fact just a flat
file containing the serialized array of coverage data that xdebug provides.
The php fileset given inside the coverage-setup specifies the PHP files to create collect
coverage information for.
The phpunit2 (the 2 being a reminisence of PHPUnit2, but it also works for PHPUnit3) tag with the codecoverage="true" runs a batch of unit tests that
are given by the fileset tests. For each file in this fileset, all tests will be run (and coverage
statistics be collected, of course).
Now the coverage-report converts the coverage database that was previously
set up to the given outfile to XML format, in our example overage_db/coverage.xml.
This XML file is transformed to a HTML report, triggered by the report tag.
In many cases, you want phing to stop the build when unit tests fail.
You'll probably not want to create a release when you know that the code did not pass all tests.
To stop a build when a test fails, you can use the haltonfailure attribute of the
phpunit2 task. To stop a build when an error occurs, use the
haltonerror attribute.
It is important to distinguish between an error and a failure.
A failure is a failing assertion in a test. For example, if you expect a function to return true
and the return value is false, NULL, or something else, your test fails.
An error would be a PHP error that occured because there is a syntax error in one of the tested files.
When creating code coverage statistics, however, we
want to execute all unit tests, regardless of how many of them fail or have errors. It would not make much sense
to stop calculating the coverage statistics when the first test fails.
When you create a beta release of your application, you might accept failing unit tests, but probably no errors, as
you wouldn't want your application to crash due to a PHP error at runtime.
Gathering coverage statistics with xdebug
There are no php.ini settings that control the creation of code coverage statistics. Gathering
of the statistics is turned on and off by PHP function at runtime, allowing for a fine-grained level
of control over which pieces of code to create coverage statistics for.
To start gathering code coverage statistics, use xdebug_start_code_coverage().
To stop gathering statistics, use xdebug_stop_code_coverage().
xdebug keeps the gathered statistics in memory while the script is executed, so you
can turn on and off code coverage within a script as often as you like. To retrieve the
collected statistics as an array, use xdebug_get_code_coverage().
When using PHPUnit together with phing to run the unit tests and gather code coverage statistics,
you will never have to worry about using these functions. PHPUnit will use them for you to collect
coverage information about the tested code.
Putting it all together
When working with PHPUnit, you need to define a test suite as a collection of tests to run multiple tests with
one command.
Using phing frees you from having to deal with test suites, since all test cases in the given
file set are executed. You just create as many individual test case classes as you like,
define a fileset that includes all test cases, and have phing run all tests. It couldn't be easier.
Like PEAR and PHPUnit, phing has a command line interface. To execute a build target, run phing
with the target to run as parameter. To create the code coverage statistics, run
phing code-coverage
This makes phing execute the target, which includes running all unit tests, collecting the coverage statistics, and then creating the HTML report of the code coverage statistics, even if one or more unit tests fail.
The following screen shot shows the HTML report generated by phing. This report is located in the
coverage_result subdirectory of your build directory, with your build directory being the directory where build.xml is located.
Code coverage statistics can help you find parts of an application
that are never executed, respectively tested.
Do not over-estimate the value of code coverage statistics.
Even 100% code coverage does not mean that your code is thoroughly tested, as the following example shows:

This small code snippet (which, admittedly, does not make much sense in the real world)
outputs numbers based on the values of $a and $b. The following table
shows the possible combinations of input values and the resulting output:
$a$bResult
truetrue11
truefalse12
falsetrue21
falsefalse22
You could achieve 100% code coverage just by testing the combinations (true, true) and (false, false).
In real code, there are often just too many execution paths to test all combinations. What is more,
defensive code often has handlers for error conditions "that should never occur". It does not make
much sense to write complex test code just to test such a rare condition.
Code coverage statistics are a nice way of getting a feel for the test coverage of an application. But please do not
overestimate the code coverage. Even 100% code coverage does not mean that your code is bug-free or even thouroughly tested.
Still, a high code coverage certainly helps building trust in the code quality of your application.
More important than achieving a high percentage of code coverage is the fact that the statistics show you
potentially critical, and untested parts of the code. These are the parts you should focus on, either by creating
tests for them, or by removing them because they are obsolete.
Conclusion
These five articles should have given you some good reasons
to start using xdebug, which really is a very good tool. I could not do without xdebug on my development system,
even if it was only for the var_dump() output and the stack traces in error messages alone.
This article ends our series of xdebug articles, so I would like to thank you for joining in.
I hope you have enjoyed reading the articles as much as I enjoyed writing them.
Very special thanks to Derick Rethans for creating and maintaining xdebug.
And to you: Happy coding - and since you have become an xdebug user now, please remember to send a postcard to Derick.
About the author:
Stefan Priebsch has been solving IT problems for over 25 years.
He is founder and CEO of e-novative GmbH, one of the first German IT
consultancies offering PHP-based solutions.
Stefan holds a university degree in computer science,
is an internationally recognized PHP expert, book author, trainer and consultant.
You can reach him at stefan.priebsch@e-novative.de.