Improve quality with code coverage tools

From SwinBrain

Testing is important (just in case you missed it the first few times we stated this). But, once a development team gets started it is hard to keep track of what is being testing and what is not being tested esp. as the code base starts to grow. This is especially hard as developers may not always write the unit tests before writing out the actual code.

Code coverage tools help developers identify gaps by highlighting code that is not being tested. They can also show the total number of tests that hit a block of code. Though this does not singularly improve the quality of the code, this can be used to ensure that the core test cases 'intentions' provide sufficient coverage of the code. When combined with metrics and style checking tools, code coverage tools can add that extra dimension by helping developers improve the quality of their code.

Contents

Code Coverage Concepts

A code coverage tool will, in most cases, print a report similar to the one below:

EMMA v2.0.4015 (stable) report, generated Sat May 15 12:02:28 CDT 2004]
------------------------------------------------------------------------------- 
OVERALL COVERAGE SUMMARY:

[class, %]	[method, %]		[block, %]		[line, %]		[name]
85%  (157/185)! 65%  (1345/2061)!	60%  (44997/74846)!	64%  (8346.3/13135)!	all classes

OVERALL STATS SUMMARY:

total packages:	8
total classes:	185
total methods:	2061
total executable files:	62
total executable lines:	13135
 
COVERAGE BREAKDOWN BY PACKAGE:

[class, %]	[method, %]	[block, %]		[line, %]		[name]
25%  (1/4)!	25%  (3/12)!	40%  (3012/7446)!	25%  (3/12)!   com.sun.tools.javac.v8.resources
94%  (16/17)!	49%  (41/83)!	48%  (1111/2292)!	45%  (201.1/450)!	com.sun.tools.javac.v8
88%  (45/51)!	61%  (242/397)! 54%  (3070/5729)!	52%  (809.6/1563)!	com.sun.tools.javac.v8.tree
83%  (19/23)!	60%  (134/224)! 54%  (2746/5063)!	56%  (580.1/1041)!	com.sun.tools.javac.v8.util
100% (1/1)	40%  (2/5)!	58%  (25/43)!		49%  (5.9/12)!	      com.sun.tools.javac
77%  (33/43)!	59%  (310/529)! 60%  (10584/17674)!	61%  (2077.2/3396)!  com.sun.tools.javac.v8.code
91%  (39/43)!	75%  (521/698)	66%  (19701/29863)!	70%  (3606.9/5138)!  com.sun.tools.javac.v8.comp
100% (3/3)	81%  (92/113)	70%  (4748/6736)!	70%  (1062.4/1523)!  com.sun.tools.javac.v8.parser

The data shown above is generated by a tool observed the code when the 'JUnit test cases' are being run. By scanning the data above, you can get a feel for which classes are being tested properly and which ones are not (e.g. Only 25% of the class com.sun…resources has been touched by the test cases while others are much better. The tools also gives an indication at the method and line level as well). The detailed report (in HTML format) will show this information in more depth.

How do these tools work?

These tools typically work through a process of program instrumentation.
Instrumentation illustration
The tool takes the program and inserts instructions that are used to monitor which paths are taken during execution. This instrumentation is performed either on the program's source code, or on its compiled code. Regardless of the input, the output is an "instrumented program" containing the original program along with additional probes. In the case of code coverage these probes will report when they are execute.
Executing the instrumented program
If you want to report the execution of the paths in the program, then probes are inserted into each branch of the program.

At the end of the execution the reports from the probes can be analysed to determine which paths were taken during the execution of the program. If we know the total number of paths in the program we can then generate percentage of paths covered in that execution of the program.

Code coverage tools for Java

In most cases, the tool will take the byte-code and will instrument that. Instrumentation of code will generally involve inserting some special blocks of code to see if a method in a class is being called. Once instrumented, the test-cases are run, and all of the metrics are collected by the tool. These are then summarised as seen the report above. Most of the tools show this a lot of clearly by indicating the exact set of lines that have not been covered by the test.

There are two types of instrumentation, "On-the-fly instrumentation" and "Pre-instrumentation". In the first case, the code is instrumented when executed and data is collected during execution. The report is created when the application is shut-down. In the second type, the class files are instrumented first and updated class files with instrumented code is written back, after this process execution takes place. The second approach is common for code that is run inside application server (like Tomcat/JBoss etc) and also useful when the report is generated during an automated build/test cycle.

There are a large number of commercial and free tools that can perform code coverage checks and generate quite nice reports. Two free (but powerful) tools are:

  1. Emma
  2. Cobertura

The reports generated look slightly different, but both offer similar features. Emma allows for on-the-fly instrumentation which is handy when developers are interested in getting some feedback on basic exploratory testing. Cobertura has all of the features, but provides some feedback on the complexity of the code being tested.

Both tools can be integrated into the standard build process by adding extra sections to the 'Ant' build script.

Code coverage tools for .NET

.NET can be instrumented in a similar way to Java. In this case the assemblies IL can be instrumented in order to insert the required probes. Strongly named assemblies present a challenge as the instrumented program will need to be signed in order to be a strongly named assembly.

Code coverage in Visual Studio .NET 2005
Visual Studio Team Test and Visual Studio Team Developer (both part of the Visual Studio .NET 2005 Team Suite) include tools to perform code coverage analysis. Code coverage can be turned on in the Test Run Configuration, you get to this via the Test then Edit Test Run Configurations menus. The image shown here illustrates the code coverage tool. The bottom of the screen shows the unit tests that were run, and the percentage of code covered by running these tests. The top of the screen shows the code, with the green bars indicating the code covered by the tests, red bars indicating the code missed. Using Team Build, unit tests and code coverage can become part of the team build process.

Further Information

For further information on this topic see:

[edit]Links: Topic Brains
Return to Swin Brain

<taglist>[code coverage, testing, java, .net, programming] Covers some of the basic details related to code coverage and provides some examples from both the Java and .NET worlds. <taglist></taglist>