Continuous Code Coverage with gcc, googletest, and Hudson

Continuous Code Coverage with gcc, googletest, and Hudson

Heres a little recipe for getting code coverage metrics for your C++ unit tests using gcc, googletest and hudson.

Suppose we have a little class that we’d like to get under continuous integration:


#ifndef PROGRESSSTATE_H_
#define PROGRESSSTATE_H_

class ProgressState {
public:
  ProgressState(unsigned int target);
  virtual ~ProgressState();

  unsigned int getPercentage();
  void setValue(int value);
private:
  unsigned int m_value;
  unsigned int m_target;
};

#endif /* PROGRESSSTATE_H_ */

And here is the implementation:


#include "ProgressState.h"

ProgressState::ProgressState(unsigned int target)
 : m_value(0), m_target(target)
{}

ProgressState::~ProgressState(){}

void ProgressState::setValue(int value)
{
  if(value < 0){
    m_value = 0;
  }
  else if(value > (int)m_target){
    m_value = m_target;
  }
  else{
    m_value = value;
  }
}

unsigned int ProgressState::getPercentage()
{
  return m_value * 100 / m_target;
}

Continuous Build

First we must install Hudson on the build machine.  Hudson is available for most operating systems and it is pretty easy to install.  Once hudson is up and running we will need to install these plugins from our hudson management console:

  • Git Plugin: To poll the git repository for your code
  • Cobertura Plugin: To gather the test coverage results a publish them with the build

The next step is to create a hudson job for building our project.

“]New Job [Hudson]

New Job [Hudson

Then we need a way for Hudson to get access to the source code.  The typical way to do this is to get hudson to poll from your version control system.  Using git this is pretty easy.

 

HudsonGitConfiguration

HudsonGitConfiguration

One thing to remember is to set up the git user account for the hudson server.  To do this in Ubuntu you need to issue the following commands:

mike@mike-laptop:~$  sudo -s -u hudson
hudson@mike-laptop:~$ cd  /var/lib/hudson/jobs/coverage/workspace
hudson@mike-laptop:/var/lib/hudson/jobs/coverage/workspace$  ls -al
total 20
drwxr-xr-x 5 hudson nogroup 4096 2010-05-31 22:33  .
drwxr-xr-x 4 hudson nogroup 4096 2010-05-31 22:33 ..
drwxr-xr-x  3 hudson nogroup 4096 2010-05-31 22:33 Debug
drwxr-xr-x 8 hudson  nogroup 4096 2010-05-31 22:33 .git
-rw-r--r-- 1 hudson nogroup 0  2010-05-31 22:33 README
drwxr-xr-x 2 hudson nogroup 4096 2010-05-31  22:33 src
hudson@mike-laptop:/var/lib/hudson/jobs/coverage/workspace$
hudson@mike-laptop:/var/lib/hudson/jobs/coverage/workspace$ git  config user.email "some@email.com"
hudson@mike-laptop:/var/lib/hudson/jobs/coverage/workspace$  git config user.name "hudson"
hudson@mike-laptop:/var/lib/hudson/jobs/coverage/workspace$

Generating the coverage statistics with gcov

To get gcc to instrument the generated binary with coverage and profiling (necessary for branch stats) code we must provide the compile with these two additional options: -fprofile-arcs -ftest-coverage. And we must link the final executable with -lgcov. Now when the coverage test executable is run it will output .gcda and .gcno files with the coverage statistics.  These settings are set in the makefiles for the project.

Write some tests

Breaking from TDD conventional wisdom for the purposes of this article, lets write some tests for the production code we already have.

#include  <gtest/gtest.h>
#include "ProgressState.h"

TEST(ProgressStateTest,  zeroValueOfValidTargetIsZeroPercent)
{
  ProgressState  progress(100);
  progress.setValue(0);
  ASSERT_EQ((unsigned int) 0,  progress.getPercentage());
}

TEST(ProgressStateTest,  negativeValueOfValidTargetIsZeroPercent)
{
  ProgressState  progress(100);
  progress.setValue(-100);
  ASSERT_EQ((unsigned int)  0, progress.getPercentage());
}

TEST(ProgressStateTest,  valueEqualTargetIsHundredPercent)
{
  ProgressState progress(200);
  progress.setValue(200);
  ASSERT_EQ((unsigned int) 100,  progress.getPercentage());
}

This looks like a pretty good start, right?  Now lets run the tests and collect the statistics in Hudson.  To do this we need to add a build step to our Hudson job.

The first three lines of the command simply execute the build.  The command on line 4 executes the binary test application we have built, and outputs the test result summary to a junit format XML file.

The final two commands are where the magic is.  This executes the gcovr script, a handy python script that converts the gcov output to a Cobertura-style XML file.

Then we have to tell hudson to search the build workspace for the junit and coverage xml files as a post-build action.  Now when we run the build we get nice overview charts trending out unit test results and code coverage.

Then when we drill down into the specific source file we can see quite clearly that we have missed a test scenario.

Links

Hudson Continuous Integration Server:

http://hudson-ci.org/

All the source code and makefiles are available publicly on my github account:

http://github.com/meekrosoft/coverage

William E. Hart’s Blog post on gcovr:

http://wehart.blogspot.com/2009/07/summarizing-gcov-coverage-statistics.html

About these ads

37 Comments

Filed under coverage, software, testing

37 responses to “Continuous Code Coverage with gcc, googletest, and Hudson

  1. j2

    Works great! Thanks a lot!

    Easily adapted this to work within our SCons build.

  2. Scott

    Thanks for posting this, it worked the first time!
    I’m using cppunit, but the rest of my setup is identical.

  3. meekrosoft

    Great, I’m glad it worked well for you both!

  4. Jose A.

    I tried it but I can’t get any information with gcovr. Dou you have some suggestion ?

  5. Jose A.

    The command is ‘ gcovr -r . -v ‘
    and I get the follow output:
    Scanning directory . for gcda files…
    Found 6 files
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/ftest.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/fractiontest.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/Fraction.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest’
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux/ftest.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux/fractiontest.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’
    Running gcov: ‘gcov /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux/Fraction.gcda –branch-counts –branch-probabilities –preserve-paths –object-directory /home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’ in ‘/home/jnieto/.jenkins/jobs/FractionTest/workspace/view_fraction_jenkins/Unix-vob/src/PrjFracTest/linux’
    Gathered coveraged data for 0 files
    ——————————————————————————
    File Lines Exec Cover Missing
    ——————————————————————————
    ——————————————————————————
    TOTAL 0 0 –%
    ——————————————————————————

  6. Jose A.

    I solved the problem, It was a problem with the Operating System language. gcovr doesn’t work in Spanish.
    One question, I cannot get to source code of file, Do you know what it could be the problem ?
    thanks.

    • meekrosoft

      Sorry Jose, I don’t know what that could be. If you find find out the solution please let me know. Good luck!

      • Jose A.

        OK. Problem solve. It was a problem about the relative paths. Cobertura didn’t find the path to source files.
        Thank you.

  7. After some issues I managed to get this working with my C++/Qt project but the code coverage results don’t seem particularly useful because it doesn’t think I have coverage on lines that allocate memory on the heap using new or new[]. Is there a way to obtain more useful coverage information for my application using gcov or another open source tool that I can integrate with Hudson like this?

  8. Pingback: Diverse läsning | La Vie est Belle

  9. Roman

    Thank you! It works great with QTestLib too.

  10. We are mandated to use Bullseye for code coverage rather than gcov,
    I know that it can output results in csv, xml or html but does any one have any information using it with husdon/jenkins?

    • meekrosoft

      I’m not sure about Bullseye as they don’t seem to display any technical specs on their product website. If the coverage results come in csv or xml it should be pretty simple to create a translator to coburtura format. Since there is an html output there is also the simple option of publishing this using the HTML publisher plugin: https://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin

      • Cheers for the reply. We have the report on the page type approach as a back up we would like to utilise the output in a more integrated manner. I haven’t played around with coburtura yet, or even seen the expected input file format yet; but yes I think thats the easiest approach if there is not a dedicated Bullseye plugin.

  11. Thomas richard

    Very useful. However I had to add a few things to make it work using out of source compilation (with cmake)

    First of all when having a project on unix you have to insure that your project workspace does not contain spaces (there is an advanced options in my version of jenkins to setup the project workspace “./workspace/projectname” works fine – or you can just avoid spaces in the project name).

    You don’t need to set gcovr to be executable every-time, if it’s in your /usr/bin directory and already executable like me it works fine.

    So the project is set up as follow:
    jenkins_proj_root/project/src are the sources for project
    jenkins_proj_root/project/unit_tests are the unit tests sources
    jenkins_proj_root/build/debug is the out of source binary directory generated by cmake (here for debug – I also have release)

    To run gcovr, add a shell execution:
    PROJ_ROOT=`pwd` # to get project root – hence no spaces in workspace
    cd build/debug/project_unit_tests_dir # go to unit test bin dir
    ./unit_tests –gtest_output=xml:../unit_tests.xml # run unit tests
    gcovr -x -e “unit_tests/*” –root=${PROJ_ROOT}/project/ CMakeFiles/project_unit_tests.dir > coverage.xml

    -x for xml output
    -e to exclude code from coverage (here the unit_tests)
    –root to specify where are the files (it had to be a global path to be available in Jenkins – hence the pwd)
    running from the CMake files dir where are the gcov files.

  12. Khurram

    Very nice!! Can we get code coverage using g++ as well?

  13. Steve

    Just a note for a problem that plagued me with Jenkins. I found that I had to use -r to specify the full path for the pwd – and it had to have a trailing “/” – until I discovered this, Cobertura (frustratingly) failed to display annotated source.

  14. Cathy

    I’m currently trying to use Cobertura and LTP GCOV (LCOV) in Hudson to generate results from one set of source codes. The biggest issue I’ve seen is that the results are very different depending on what tool was used.

    Does anyone have an idea why the data/output is different and what would be better tool to use with Hudson?

  15. naushad

    The cobertura does not take care of mocked functions and its report assumes that mocked function should also be covered in gtest. Is there a way, we can exclude mocked functions in generating the code coverage report.

    • meekrosoft

      This is not a Coburtura thing, but more of a gcov thing. If you really want to ignore coverage for test data you will need to filter it out manually I think.

  16. zero

    hi , i have launched hudson and create coverage project followed by your blog on ubuntu . I can build the project successfully . But no code coverage .(hudson gcovr ubuntu shell)

  17. Jack

    I like this setup, but am having one problem. I can’t find a way to show zero coverage stats for files that are NOT executed as part of the tests. These source files will have associated .gcno files, but because they are not ran no .gcda files are produced and and gcovr doesn’t show them in the results.

    Has anyone found a way to get around this (like perhaps creating a pre-processing step that generates zero initalized .gcda files?) I’ve tried using touch to create such files but they’re not accepted as valid by gcov.

    • meekrosoft

      Ah, that is a very good point. I don’t know the answer to your question, but if you find out please let me know.

      • Jack

        I couldn’t find a way to easily initialize empty gcda files. I ended up hacking the gcovr script and adding an “–estimate-uncovered-files” option. When enabled gcovr looks at each of the files with gcno but no gcda, estimates line counts (making simple attempts to avoid blank and comment lines), and then adds the data to the internal covdata dict before printing output. It’s only about a dozen lines…

        The code indicates that at some point an effort was made to process stand alone gcno files, but perhaps wasn’t finished?

        Unfortunately my code is not generic enough to submit as a patch for gcovr (assumes certain extensions for source/headers, certain conventions for comments etc.), but it works for my purposes.

  18. Greetings from Carolina! I’m bored to tears at work so I decided to browse your blog on my iphone during lunch break. I really like the knowledge you provide here and can’t wait to take a look when I get home. I’m amazed at how fast your blog loaded on my phone .. I’m not even using WIFI, just 3G .. Anyhow, awesome blog!

  19. In the include section of the makefile on github, you include a file named subdir.mk (not the same one as in src/subdir.mk). However, you have not pushed this file to the repository. What’s in this file?

    I am having issues with hudson compiling my code with all compile options that I’ve specified (like -std=c++0x). Right now it’s only using -c and -o

    (This is an awesome post btw, best one on code coverage with gcc, gtest and hudson out there!)

    • meekrosoft

      Sorry, I don’t have that sourcecode anymore because of a hard disk failure :-( At any rate, the subdir.mk file is autogenerated by eclipse so if you create an empty c++ project and add the source files you should get it to build. Let me know if this doesn’t work for you.

  20. Hello
    I have some issues with -lgtest and -lgtest_main flags.
    How fix?

    Additional information
    g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
    GNU ld (GNU Binutils for Ubuntu) 2.22

  21. meekrosoft

    Hi NaneNare,
    It should be that g++ can’t find the libraries – are you sure they are available either in the build directory or the library pat?
    Cheers,
    Mike

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s