Automated Test Coverage Checks with Travis, PHPUnit for Github Pull Requests

Code quality

Github is fun, and it is a great place for showing off our coding skills and building great tools for our next great project!
But what if we want to also ensure that our tool works perfectly?
Obviously, perfection is not a term of the developer world. There is always a flaw waiting for us behind the corner, but we can do our best to try to avoid it by testing our code, and testing it well.

Code coverage

Code coverage is one of the metrics we can get from PHPUnit tests, and it is a strong indicator of how hard we worked on testing our application. It is not an universal metric to define if our code works, and there is tons of examples of how your code can still be buggy even if every line of code was executed at least once. It should anyway not be ignored, and it is up to us to decide how hard we want to test each part of our application.

That's why I came up with the idea of automating a simple coverage check on a library I was working on.

The script

What we basically want to do is to automate a very simple check that verifies that our test suite covered at least a given percentage of the ] ELOC. The script should exit with an exit code different from 1 if the check wasn't successful, thus being recognized by our test runner as a failure.

I will use Travis-CI for my examples, but what I am going to show can be easily integrated also in other Continuous Integration environments.

PHPUnit setup and clover.xml

PHPUnit is able to generate log files in an XML format called clover. You will need to setup PHPUnit as following to generate a correct clover report:

<?xml version="1.0"?>
<!-- works fine with PHPUnit-3.6.10 -->
<phpunit>
    <!-- you can keep your own options in these elements -->
    <filter>
        <whitelist addUncoveredFilesFromWhitelist="true">
            <!-- this is the path of the files included in your clover report -->
            <directory suffix=".php">./src</directory>
        </whitelist>
    </filter>
    <logging>
        <!-- and this is where your report will be written -->
        <log type="coverage-clover" target="./clover.xml"/>
    </logging>
</phpunit>

Now let's try to run our test suite... you will get a clover.xml file like following:

<?xml version="1.0" encoding="UTF-8"?>
    <coverage generated="1345080270">
        <project timestamp="1345080270"></project>
            <file name="/home/ocramius/Desktop/ZendSkeletonApplication/vendor/rwoverdijk/assetmanager/src/ClassName.php">
                <class name="ClassName" namespace="global">
                    <metrics methods="1" coveredmethods="0" conditionals="0" coveredconditionals="0" statements="114" coveredstatements="0" elements="115" coveredelements="0"/>
                </class>
                <line num="12" type="method" name="testMethod" crap="2" count="0"/>
                <line num="15" type="stmt" count="0"/>
                <line num="16" type="stmt" count="0"/>
                <line num="17" type="stmt" count="0"/>
                <line num="18" type="stmt" count="0"/>
                <line num="19" type="stmt" count="0"/>
                <line num="20" type="stmt" count="0"/>
                <-- ... [continues] ... -->

You can read more about this on the PHPUnit docs.

We now want to check this log in our script. What seems interesting for us is coverage -> project -> file -> class -> metrics -> elements|coveredelements, which gives us the number of elements that can could be executed and the number of elements that have been executed.

The PHP script

And here's the PHP script that will handle all this. It is just using SimpleXMLElement and an XPath expression to find elements relevant to us and extract them from the XML.

<?php
// coverage-checker.php
$inputFile  = $argv[1];
$percentage = min(100, max(0, (int) $argv[2]));

if (!file_exists($inputFile)) {
    throw new InvalidArgumentException('Invalid input file provided');
}

if (!$percentage) {
    throw new InvalidArgumentException('An integer checked percentage must be given as second parameter');
}

$xml             = new SimpleXMLElement(file_get_contents($inputFile));
$metrics         = $xml->xpath('//metrics');
$totalElements   = 0;
$checkedElements = 0;

foreach ($metrics as $metric) {
    $totalElements   += (int) $metric['elements'];
    $checkedElements += (int) $metric['coveredelements'];
}

$coverage = ($checkedElements / $totalElements) * 100;

if ($coverage < $percentage) {
    echo 'Code coverage is ' . $coverage . '%, which is below the accepted ' . $percentage . '%' . PHP_EOL;
    exit(1);
}

echo 'Code coverage is ' . $coverage . '% - OK!' . PHP_EOL;

The script basically asks for two input arguments. One is the clover file path and one is the minimum percentage of code coverage we want. Usage is like following:

~$ php coverage-checker.php clover.xml 80
Code coverage is 74.32%, which is below the accepted 80%

~$ php coverage-checker.php clover.xml 70
Code coverage is 74.32% - OK!

Putting it in our CI environment

And here's the simple .travis.yml configuration to make everything work!

language: php

php:
  - 5.3
  - 5.4

script:
  - phpunit
  - php coverage-checker.php clover.xml 75

Was it hard? Quite impressive for such a simple script combined with what PHPUnit can do! If you don't want to try it out yourself, here's a link to a recent build with 100% code coverage: http://travis-ci.org/#!/Ocramius/AssetManager/jobs/2137625/L83 . Now you don't need to ask anyone to write tests for the features they just wrote. Just let the tests fail and they'll see it directly! :-)

Enjoy!

Tags: phpunit, code, coverage, travis, travis, continuos integration, testing

Marco Pivetta (Ocramius)