Why code coverage for Behat?
PHPUnit has built-in several options for generating code coverage data and reports. Behat doesn’t. As Konstantin Kudryashov (@everzet) points out in an issue asking for code coverage options in Behat:
Code coverage is controversial idea and code coverage for StoryBDD framework is just nonsense. If you’re doing code testing with StoryBDD – you’re doing it wrong.
He’s right I guess. The main issue is that StoryBDD isn’t about code, so it doesn’t make sense to calculate code coverage for it. Furthermore, the danger of collecting code coverage data and generating coverage reports is that people will start using it as a metric for code quality. And maybe they’ll even set management targets based on coverage percentage. Anyway, that’s not what this article is about…
In my Advanced Application Testing workshop I just wanted to collect code coverage data to show how different types of tests (system, acceptance, integration and unit) touch different parts of the code. And how this evolves over time, when we’re replacing parts of the system under test, or switch test types to achieve shorter feedback loops, etc.
The main issue is that, when it comes to running our code, Behat does two things: it executes code in the same project, and/or (and this complicates the situation a bit) it remotely executes code when it’s using Mink to talk to a web application running in another process.
This means that if you want to have Behat coverage, you’ll need to do two things:
- Collect local code coverage data.
- Instruct the remote web application to collect code coverage data itself, then fetch it.
Behat extensions for code coverage
For the workshop, I created two Behat extensions that do exactly this:
The second one makes use of an adapted version of the
LiveCodeCoverage tool I published earlier.
You have to enable these extensions in your
default: extensions: BehatMinkExtension: # ... BehatLocalCodeCoverageLocalCodeCoverageExtension: target_directory: '%paths.base%/var/coverage' BehatRemoteCodeCoverageRemoteCodeCoverageExtension: target_directory: '%paths.base%/var/coverage' suites: acceptance: # ... local_coverage_enabled: true system: mink_session: default # ... remote_coverage_enabled: true
Local coverage doesn’t require any changes to the production code, but remote coverage does: you need to run a tool called
RemoteCodeCoverage, and let it wrap your application/kernel in your web application’s front controller (e.g.
use LiveCodeCoverageRemoteCodeCoverage; $shutDownCodeCoverage = RemoteCodeCoverage::bootstrap( (bool)getenv('CODE_COVERAGE_ENABLED'), sys_get_temp_dir(), __DIR__ . '/../phpunit.xml.dist' ); // Run your web application now... // This will save and store collected coverage data: $shutDownCodeCoverage();
From now on, a Behat run will generate a coverage file (
./var/coverage for every suite that has coverage enabled (the name of the file is the name of the suite).
The arguments passed to
RemoteCodeCoverage::bootstrap() allow for some fine-tuning of its behavior:
- Provide your own logic to determine if code coverage should be enabled in the first place (this example uses an environment variable for that). This is important for security reasons. It helps you make sure that the production server won’t expose any collected coverage data.
- Provide your own directory for storing the coverage data files.
- Provide the path to your own
phpunit.xml(.dist)file. This file is used for its code coverage filter configuration. You can exclude
vendor/code for example.
Combining coverage data from different tools and suites
If you also instruct PHPUnit to generate “PHP” coverage files in the same directory, you will end up with several
.cov files for every test run, one for every suite/type of test.
vendor/bin/phpunit --testsuite unit --coverage-php var/coverage/unit.cov vendor/bin/phpunit --testsuite integration --coverage-php var/coverage/integration.cov vendor/bin/behat --suite acceptance vendor/bin/behat --suite system // these files will be created (and overwritten during subsequent runs): var/coverage/unit.cov var/coverage/integration.cov var/coverage/acceptance.cov var/coverage/system.cov
Finally, you can merge these files using phpcov:
phpcov merge --html=./var/coverage/html ./var/coverage
You’ll get this nice HTML report and for every line it shows you which unit test or feature file “touched” this line:
I hope these extensions prove to be useful; please let me know if they are. For now, the biggest downside is slowness – running code with XDebug enabled is already slow, but collecting code coverage data along the way is even slower. Still, the benefits may be big enough to justify this.