Boost Unit Test Framework

Boost is a widely used set of C++ libraries that made writing C++ a bit easier. Linear algebra, smart pointers, you name it. It’s probably in Boost. It has a lot of batteries included.

One of those batteries is the Boost Test Library. Boost Test provides a unit test framework, which handles much of the tedious overhead (e.g., outputting errors) associated with testing code. In fact, the test framework will even give you a main() function for (nearly) free!

As discussed in class, part of writing unit tests is asserting that certain facts are true or false about the code you’re testing. For example, if you’ve written a C++ function int add(int x, int y) that simply sums two numbers, you could assert that the return value of add(3, 4) is equal to 7. If add(3, 4) is not 7, then something is wrong. Really wrong.

Boost provides some functions1 that make it easy to make assertions about our code. The provided code base has already #included all the .h files that you will need from Boost to use its unit testing framework.

Assertions

Levels

Boost test framework’s assertions come in three flavors (levels):

Level Meaning
WARN Output a warning message, but don’t count it as an error. Keep running the rest of the test case. This level can be useful for displaying warnings about low-severity issues. You won’t need this one.
CHECK Output an error message, and count it as an error. Keep running the rest of the test case. This level is useful for asserting correctness, which is what we want. We can report a problem without halting the execution of the test case.
REQUIRE Output an error message, and count it as an error. DO NOT keep running the rest of the test case. Move on to the next test case instead. This level is used when a bad assertion prevents running the rest of the test case. For example, if we need to open a file as part of our test, we might BOOST_REQUIRE(file.is_good()) to make sure it opened correctly. If our file failed to open, then we stop the test case. We wouldn’t be able to proceed, anyway!

For the first homework assignment, we’ll be using CHECK-level assertions. If, however, you feel that a WARN- or REQUIRE-level assertion is necessary, you can certainly do that. In that case, be sure to include a comment explaining why.

Assertion Tools

This is a quick summary of the tools you can use in your tests. Just replace the <level> bit with a level (WARN, CHECK, or REQUIRE). You can find more assertion tools in Boost’s documentation: http://www.boost.org/doc/libs/1_39_0/libs/test/doc/html/utf/testing-tools/reference.html.

Assertion What it does Example
BOOST_<level> Assert that a statement is true BOOST_CHECK(1 == 1)
BOOST_<level>_EQUAL Assert that two values are equal BOOST_CHECK_EQUAL(1, 1)
BOOST_<level>_GE Assert that the left value is greater than or equal to the right value BOOST_CHECK_GE(1.1, 1.0)
BOOST_<level>_GT Assert that the left value is strictly greater than the right value BOOST_CHECK_GT(1, 0)
BOOST_<level>_LE Assert that the left value is less than or equal to the right value BOOST_CHECK_LE(1.1, 1.0)
BOOST_<level>_LT Assert that the left value is strictly less than the right value BOOST_CHECK_LT(1, 0)

An Example Test Case

Below is an example of a test case for an eat member function for a Dog class. When Dog is created, its default constructor sets its weight to 10. When a Dog eats food (a string), it gains one pound for every character in the string. We can check the dog’s current weight with its getWeight() getter function. Also, a Dog is always full right after eating, which we can check with its isFull() function.

// Test that the dog's default weight is set properly.
BOOST_AUTO_TEST_CASE( test_default_weight )
{
  Dog d;
  BOOST_CHECK_EQUAL(d, 10);
}

// Test that the dog gains weight when it eats and gets full
BOOST_AUTO_TEST_CASE( test_eating )
{
  Dog d;
  int current_weight = d.getWeight();
  d.eat("banana");

  // The length of "banana" is 6. Should have gained 6 pounds.
  BOOST_CHECK_EQUAL(d.getWeight(), current_weight + 6);

  // Make sure that the dog is full
  BOOST_CHECK(d.isFull());
}

Notice the syntax for test cases. Boost uses some more advanced C++ features (macros) to make it easier to write tests. You’ll notice that every test case in the assignment starter code uses this format. The important thing to take away is that the code for the test case goes between the curly braces for the test case. In that space, you’ll write C++ code that uses assertions (e.g., BOOST_CHECK) to assert that the code is working as expected.

Code Coverage

A code coverage report tells us which lines of code were executed during a run. Code coverage can give us a hint as to whether or not our unit tests actually test all of the code it’s supposed to. If, for example, a coverage report shows that your unit tests only used 50% of the lines of C++ in a source file… then you should probably write some more unit tests or figure out why your tests aren’t reaching those lines.

In the following subsection, we will discuss a C++ test coverage tool called gcov. You will not be required to invoke gcov directly for homework 1. The starter code repository includes a script that will do it for you. For more details on how to run that script, check out the README.md file in the starter repository.

This page is simply for your own knowledge. You should still read it.

gcov

In addition to giving the world g++, the kind people of GNU created gcov – a test coverage program. To use gcov, you have to compile your program with a couple of additional libraries.

Example From First Lecture New!

The example covered in the very first lecture talked about testing a zip_code_is_valid() function. Without looking at the function’s implementation, we wrote our test cases based on the behavior described by the comment above the function’s prototype (in the .h file). We then looked at the tests’ code coverage for insights into how thorough the tests were. The code for this example can be found here.

Another Example Program

Let’s look at another example. Here’s a file called main.cpp:


/**
* \file main.cpp
*
* An example program
*/

#include <iostream>

using namespace std;

int main()
{
  int i;
  for (i = 0; i < 10; i++)
  {
    if (true)
      cout << "Hello world!" << endl;
  }

  // It's 10, not 50
  if (i == 50)
    cout << "Impossible!" << endl;

  return 0;
}

If we wanted to compile main.cpp and determine its coverage, we would need to do the following:

# Compile the program (generates main.gcno data file)
$ fg++ -o my_program --profile-arcs --test-coverage main.cpp

# Run it (generates main.gcda file)
$ ./my_program

# Generate coverage reports (uses main.gcno and main.gcda)
$ gcov main.cpp
File 'main.cpp'
Lines executed:85.71% of 7
Creating 'main.cpp.gcov'

...

A few things to note:

  1. The --profile-arcs and --test-coverage flags tell g++ to generate some extra data files that gcov needs in order to create coverage reports.
  2. You have to run the program (./my_program) in order to generate even more data files needed by gcov.
  3. gcov will print the coverage percentages to to console when you run it. It will also create several *.gcov files that show how many times each line in the file was run

For our main.cpp, gcov will create main.cpp.gcov, which shows how many times each line was run.

      -:    0:Source:main.cpp
    -:    0:Graph:main.gcno
    -:    0:Data:main.gcda
    -:    0:Runs:1
    -:    0:Programs:1
    -:    1:/**
    -:    2: * \file main.cpp
    -:    3: *
    -:    4: * An example program
    -:    5: */
    -:    6:
    -:    7:#include <iostream>
    -:    8:
    -:    9:using namespace std;
    -:   10:
    1:   11:int main()
    -:   12:{
    -:   13:  int i;
   11:   14:  for (i = 0; i < 10; i++)
    -:   15:  {
    -:   16:    if (true)
   10:   17:      cout << "Hello world!" << endl;
    -:   18:  }
    -:   19:
    -:   20:  // It's 10, not 50
    1:   21:  if (i == 50)
#####:   22:      cout << "Impossible!" << endl;
    -:   23:
    1:   24:  return 0;
    3:   25:}


We can tell a few things from this file. Briefly…

  1. Line 11 tells us that we ran main() one time.
  2. We ran line 17 10 times.
  3. We executed the for statement 11 times
    • Don’t forget that we have to run the i < 10 a total 11 times: 10 of those times it returns true, and one of those times it returns false to signal the end of the loop.
  4. We never ever ran line 22. That’s the significance of the #####.

As you can see, there’s quite a bit of information packed into that coverage report. If it looks like one line of code is running too much, maybe it needs to be optimized!

  1. Well, actually, they’re macros. You call them like functions, though. If you want to, go search for “C++ macro” sometime. Don’t get hung up on them though. The macros that we care about using are syntactically similar to calling a function.