     [file version.texi not included]

Check
*****

This manual is for Check (version VERSION, UPDATED), a unit test framework for C programs.

Copyright © 2001--2006 Arien Malec, Chris Pickett, Fredrik Hugosson, and Robert Lemmen.

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License.''

Please send corrections to this manual to check-devel@lists.sourceforge.net.  We'd prefer it if you can send a unified diff (diff -u) against the doc/check.texi file that ships with Check, but something is still better than nothing if you can't manage that.

Introduction
************

Check is a unit testing framework for C.  It was inspired by similar frameworks that currently exist for most programming languages; the most famous example being JUnit (http://www.junit.org) for Java. There is a list of unit test frameworks for multiple languages at http://www.xprogramming.com/software.htm.  Unit testing has a long history as part of formal quality assurance methodologies, but has recently been associated with the lightweight methodology called Extreme Programming.  In that methodology, the characteristic practice involves interspersing unit test writing with coding (``test a little, code a little'').  While the incremental unit test/code approach is indispensable to Extreme Programming, it is also applicable, and perhaps indispensable, outside of that methodology.

The incremental test/code approach provides three main benefits to the developer:

 1. Because the unit tests use the interface to the unit being tested, they allow the developer to think about how the interface should be designed for usage early in the coding process.

 2. They help the developer think early about aberrant cases, and code accordingly.

 3. By providing a documented level of correctness, they allow the developer to refactor (see http://www.refactoring.com) aggressively.

That third reason is the one that turns people into unit testing addicts.  There is nothing so satisfying as doing a wholesale replacement of an implementation, and having the unit tests reassure you at each step of that change that all is well.  It is like the difference between exploring the wilderness with and without a good map and compass: without the proper gear, you are more likely to proceed cautiously and stick to the marked trails; with it, you can take the most direct path to where you want to go.

Look at the Check homepage for the latest information on Check: http://check.sourceforge.net.

The Check project page is at: http://sourceforge.net/projects/check/.

Unit Testing in C
*****************

 C unit testing

The approach to unit testing frameworks used for Check originated with Smalltalk, which is a late binding object-oriented language supporting reflection.  Writing a framework for C requires solving some special problems that frameworks for Smalltalk, Java or Python don't have to face.  In all of those language, the worst that a unit test can do is fail miserably, throwing an exception of some sort.  In C, a unit test is just as likely to trash its address space as it is to fail to meet its test requirements, and if the test framework sits in the same address space, goodbye test framework.

To solve this problem, Check uses the fork() system call to create a new address space in which to run each unit test, and then uses message queues to send information on the testing process back to the test framework.  That way, your unit test can do all sorts of nasty things with pointers, and throw a segmentation fault, and the test framework will happily note a unit test error, and chug along.

The Check framework is also designed to play happily with common development environments for C programming.  The author designed Check around Autoconf/Automake (thus the name Check: make check is the idiom used for testing with Autoconf/Automake), and the test failure messages thrown up by Check use the common idiom of filename:linenumber:message used by gcc and family to report problems in source code.  With (X)Emacs, the output of Check allows one to quickly navigate to the location of the unit test that failed; presumably that also works in VI and IDEs.

Other Frameworks for C
======================

The authors know of the following additional unit testing frameworks for C:

GNU Autounit
     Much along the same lines as Check, including forking to run unit tests in a separate address space (in fact, the original author of Check borrowed the idea from GNU Autounit).  GNU Autounit uses GLib extensively, which means that linking and such need special options, but this may not be a big problem to you, especially if you are already using GTK or GLib.  See the GNU Autounit homepage (http://www.recursism.com/s2004/zp/products/gnu+autounit).

cUnit
     Also uses GLib, but does not fork to protect the address space of unit tests.  See the archived cUnit homepage (http://web.archive.org/web/*/http://people.codefactory.se/~spotty/cunit/).

CUnit
     Standard C, with plans for a Win32 GUI implementation.  Does not currently fork or otherwise protect the address space of unit tests. In early development.  See the CUnit homepage (http://cunit.sourceforge.net).

CppUnit
     The premier unit test framework for C++; you can also use it to test C code.  It is stable, actively developed, and has a GUI interface.  The primary reasons not to use CppUnit for C are first that it is quite big, and second you have to write your tests in C++, which means you need a C++ compiler.  If these don't sound like concerns, it is definitely worth considering, along with other C++ unit testing frameworks.  See the CppUnit homepage (http://cppunit.sourceforge.net/cppunit-wikie).

MinUnit
     A minimal set of macros and that's it!  The point is to show how easy it is to unit test your code.  See the MinUnit homepage (http://www.jera.com/techinfo/jtns/jtn002.html).

CUnit for Mr. Ando
     A CUnit implementation that is fairly new, and apparently still in early development.  See the CUnit for Mr. Ando homepage (http://park.ruru.ne.jp/ando/work/CUnitForAndo/html/).

This list was last updated in March 2006.  If you know of other C unit test frameworks, please send an email plus description to check-devel@lists.sourceforge.net and we will add the entry to this list.

It is the authors' considered opinion that forking or otherwise trapping and reporting signals is indispensable for unit testing (but it probably wouldn't be hard to add that to frameworks without that feature).  Try 'em all out: adapt this tutorial to use all of the frameworks above, and use whichever you like.  Contribute, spread the word, and make one a standard.  Languages such as Java and Python are fortunate to have standard unit test frameworks; it would be desirable that C have one as well.

Tutorial: Basic Unit Testing
****************************

This tutorial will use the JUnit Test Infected (http://junit.sourceforge.net/doc/testinfected/testing.htm) article as a starting point.  We will be creating a library to represent money, libmoney, that allows conversions between different currency types.  The development style will be ``test a little, code a little'', with unit test writing preceding coding. This constantly gives us insights into module usage, and also makes sure we are constantly thinking about how to test our code.

How to Write a Test
===================

Test writing using Check is very simple. The file in which the checks are defined must include check.h as so:

     #include <check.h>

The basic unit test looks as follows:

     START_TEST (test_name)
     {
       /* unit test code */
     }
     END_TEST

The START_TEST/END_TEST pair are macros that setup basic structures to permit testing.  It is a mistake to leave off the END_TEST marker; doing so produces all sorts of strange errors when the check is compiled.

Setting Up the Money Build
==========================

Since we are creating a library to handle money, we will first create an interface in money.h, an implementation in money.c, and a place to store our unit tests, check_money.c.  We want to integrate these core files into our build system, and will need some additional structure. To manage everything we'll use Autoconf, Automake, and friends (collectively known as Autotools) for this example.  One could do something similar with ordinary Makefiles, but in the authors' opinion, it is generally easier to use Autotools than bare Makefiles, and they provide built-in support for running tests.

Note that this is not the place to explain how Autotools works.  If you need help understanding what's going on beyond the explanations here, the best place to start is probably Alexandre Duret-Lutz's excellent Autotools tutorial (http://www-src.lip6.fr/homepages/Alexandre.Duret-Lutz/autotools.html).

The examples in this section are part of the Check distribution; you don't need to spend time cutting and pasting or (worse) retyping them. Locate the Check documentation on your system and look in the example directory.  The standard directory for GNU/Linux distributions should be /usr/share/doc/check/example.  This directory contains the final version reached the end of the tutorial.  If you want to follow along, create backups of money.h, money.c, and check_money.c, and then delete the originals.

We set up a directory structure as follows:

     .
     |-- Makefile.am
     |-- README
     |-- configure.ac
     |-- src
     |   |-- Makefile.am
     |   |-- main.c
     |   |-- money.c
     |   `-- money.h
     `-- tests
         |-- Makefile.am
         `-- check_money.c

Note that this is the output of tree, a great directory visualization tool.  The top-level Makefile.am is simple; it merely tells Automake how to process subdirectories:

     SUBDIRS = src . tests

Note that tests comes last, because the code should be testing an already compiled library.  configure.ac is standard Autoconf boilerplate, as specified by the Autotools tutorial and as suggested by autoscan.  The AM_PATH_CHECK() is the only line particular to Check see AM_PATH_CHECK.

src/Makefile.am builds libmoney as a Libtool archive, and links it to an application simply called main.  The application's behaviour is not important to this tutorial; what's important is that none of the functions we want to unit test appear in main.c; this probably means that the only function in main.c should be main() itself.  In order to test the whole application, unit testing is not appropriate: you should use a system testing tool like Autotest.  If you really want to test main() using Check, rename it to something like _myproject_main() and write a wrapper around it.

The primary build instructions for our unit tests are in tests/Makefile.am:

          [file Makefile.am not included]

TESTS tells Automake which test programs to run for make check.  Similarly, the check_ prefix in check_PROGRAMS actually comes from Automake; it says to build these programs only when make check is run.  (Recall that Automake's check target is the origin of Check's name.)  The check_money test is a program that we will build from tests/check_money.c, linking it against both src/libmoney.la and the installed libcheck.la on our system.  The appropriate compiler and linker flags for using Check are found in @CHECK_CFLAGS@ and @CHECK_LIBS@, values defined by the AM_PATH_CHECK macro.

Now that all this infrastructure is out of the way, we can get on with development.  src/money.h should only contain standard C header boilerplate:

          [file money.1.h not included]

src/money.c should be empty, and tests/check_money.c should only contain an empty main() function:

          [file check_money.1.c not included]

Create the GNU Build System for the project and then build main and libmoney.la as follows:

     $ autoreconf --install
     $ ./configure
     $ make

(autoreconf determines which commands are needed in order for configure to be created or brought up to date. Previously one would use a script called autogen.sh or bootstrap, but that practice is unnecessary now.)

Now build and run the check_money test with make check.  If all goes well, make should report that our tests passed.  No surprise, because there aren't any tests to fail.  If you have problems, make sure to see AM_PATH_CHECK.

This was tested on the i386 ``testing'' distribution of Debian GNU/Linux (etch) in March 2006, using Autoconf 2.59, Automake 1.9.6, and Libtool 1.5.22.  Please report any problems to check-devel@lists.sourceforge.net.

Test a Little, Code a Little
============================

The Test Infected (http://junit.sourceforge.net/doc/testinfected/testing.htm) article starts out with a Money class, and so will we.  Of course, we can't do classes with C, but we don't really need to.  The Test Infected approach to writing code says that we should write the unit test before we write the code, and in this case, we will be even more dogmatic and doctrinaire than the authors of Test Infected (who clearly don't really get this stuff, only being some of the originators of the Patterns approach to software development and OO design).

Here are the changes to check_money.c for our first unit test:

          [file check_money.1-2.c.diff not included]

A unit test should just chug along and complete.  If it exits early, or is signaled, it will fail with a generic error message.  (Note: it is conceivable that you expect an early exit, or a signal.  There is currently nothing in Check to specifically assert that we should expect either --- if that is valuable, it may be worth while adding to Check.)  If we want to get some information about what failed, we need to use the fail_unless() function.  The function (actually a macro) takes a first Boolean argument, and an error message to send if the condition is not true.

If the Boolean argument is too complicated to elegantly express within fail_unless(), there is an alternate function fail() that unconditionally fails.  The second test inside test_money_create above could be rewritten as follows:

     if (strcmp (money_currency (m), "USD") != 0)
       {
         fail ("Currency not set correctly on creation");
       }

There is also a fail_if() function, which is the inverse of fail_unless().  Using it, the above test then looks like this:

     fail_if (strcmp (money_currency (m), "USD") != 0,
              "Currency not set correctly on creation");

For your convenience all fail functions also accepts NULL as the msg argument and substitutes a suitable message for you.  So you could also write a test as follows:

     fail_unless (money_amount (m) == 5, NULL);

This is equivalent to:

     fail_unless (money_amount (m) == 5,
                  "Assertion 'money_amount (m) == 5' failed");

All fail functions also support varargs and accept printf-style format strings and arguments.  This is especially useful while debugging.  With printf-style formatting the message could look like this:

     fail_unless(money_amount (m) == 5,
                 "Amount was %d, instead of 5", money_amount (m));

When we try to compile and run the test suite now using make check, we get a whole host of compilation errors.  It may seem a bit strange to deliberately write code that won't compile, but notice what we are doing: in creating the unit test, we are also defining requirements for the money interface.  Compilation errors are, in a way, unit test failures of their own, telling us that the implementation does not match the specification.  If all we do is edit the sources so that the unit test compiles, we are actually making progress, guided by the unit tests, so that's what we will now do.

We will patch our header money.h as follows:

          [file money.1-2.h.diff not included]

Our code compiles now, and again passes all of the tests.  However, once we try to use the functions in libmoney in the main() of check_money, we'll run into more problems, as they haven't actually been implemented yet.

Creating a Suite
================

To run unit tests with Check, we must create some test cases, aggregate them into a suite, and run them with a suite runner. That's a bit of overhead, but it is mostly one-off.  Here's a diff for the new version of check_money.c.  Note that we include stdlib.h to get the definitions of EXIT_SUCCESS and EXIT_FAILURE.

          [file check_money.2-3.c.diff not included]

Most of the money_suite() code should be self-explanatory.  We are creating a suite, creating a test case, adding the test case to the suite, and adding the unit test we created above to the test case. Why separate this off into a separate function, rather than inline it in main()?  Because any new tests will get added in money_suite(), but nothing will need to change in main() for the rest of this example, so main will stay relatively clean and simple.

Unit tests are internally defined as static functions.  This means that the code to add unit tests to test cases must be in the same compilation unit as the unit tests themselves.  This provides another reason to put the creation of the test suite in a separate function: you may later want to keep one source file per suite; defining a uniquely named suite creation function allows you later to define a header file giving prototypes for all the suite creation functions, and encapsulate the details of where and how unit tests are defined behind those functions.  See the test program defined for Check itself for an example of this strategy.

The code in main() bears some explanation.  We are creating a suite runner object of type SRunner from the Suite we created in money_suite().  We then run the suite, using the CK_NORMAL flag to specify that we should print a summary of the run, and list any failures that may have occurred.  We capture the number of failures that occurred during the run, and use that to decide how to return.  The check target created by Automake uses the return value to decide whether the tests passed or failed.

Now that the tests are actually being run by check_money, we encounter linker errors again we try out make check.  Try it for yourself and see.  The reason is that the money.c implementation of the money.h interface hasn't been created yet.  Let's go with the fastest solution possible and implement stubs for each of the functions in money.c:

          [file money.1-3.c.diff not included]

Note that we #include <stdlib.h> to get the definition of NULL.  Now, the code compiles and links when we run make check, but our unit test fails.  Still, this is progress, and we can focus on making the test pass.

SRunner Output
==============

The function to run tests in an SRunner is defined as follows:

     void srunner_run_all (SRunner * sr, enum print_output print_mode);

This function does two things:

 1. It runs all of the unit tests for all of the test cases defined for all of the suites in the SRunner, and collects the results in the SRunner

 2. It prints the results according to the print_mode specified.

For SRunners that have already been run, there is also a separate printing function defined as follows:

     void srunner_print (SRunner *sr, enum print_output print_mode);

The enumeration values of print_output defined in Check that parameter print_mode can assume are as follows:

CK_SILENT
     Specifies that no output is to be generated. If you use this flag, you either need to programmatically examine the SRunner object, print separately, or use test logging (see Test Logging.)

CK_MINIMAL
     Only a summary of the test run will be printed (number run, passed, failed, errors).

CK_NORMAL
     Prints the summary of the run, and prints one message per failed test.

CK_VERBOSE
     Prints the summary, and one message per test (passed or failed)

CK_ENV
     Gets the print mode from the environment variable CK_VERBOSITY, which can have the values "silent", "minimal", "normal", "verbose". If the variable is not found or the value is not recognized, the print mode is set to CK_NORMAL.

With the CK_NORMAL flag specified in our main(), let's rerun make check now. As before, we get the following satisfying output:

     Running suite(s): Money
     0%: Checks: 1, Failures: 1, Errors: 0
     check_money.c:10:F:Core:test_money_create: Amount not set correctly on
     creation
     FAIL: check_money
     ==================================================
     1 of 1 tests failed
     Please report to check-devel@lists.sourceforge.net
     ==================================================

The first number in the summary line tells us that 0% of our tests passed, and the rest of the line tells us that there was one check in total, and of those checks, one failure and zero errors.  The next line tells us exactly where that failure occurred, and what kind of failure it was (P for pass, F for failure, E for error).

After that we have some higher level output generated by Automake: the check_money program failed, and the bug-report address given in configure.ac is printed.

Let's implement the money_amount function, so that it will pass its tests.  We first have to create a Money structure to hold the amount, and then implement the function to return the correct amount:

          [file money.3-4.c.diff not included]

We will now rerun make check and... what's this?  The output is now as follows:

     Running suite(s): Money
     0%: Checks: 1, Failures: 0, Errors: 1
     check_money.c:5:E:Core:test_money_create: (after this point) Received
     signal 11 (Segmentation fault)

What does this mean?  Note that we now have an error, rather than a failure.  This means that our unit test either exited early, or was signaled.  Next note that the failure message says ``after this point''; This means that somewhere after the point noted (check_money.c, line 5) there was a problem: signal 11 (a.k.a. segmentation fault).  The last point reached is set on entry to the unit test, and after every call to fail_unless(), fail(), or the special function mark_point().  For example, if we wrote some test code as follows:

     stuff_that_works ();
     mark_point ();
     stuff_that_dies ();

then the point returned will be that marked by mark_point().

The reason our test failed so horribly is that we haven't implemented money_create() to create any Money.  We'll go ahead and implement that, the symmetric money_free(), and money_currency() too, in order to make our unit test pass again:

          [file money.4-5.c.diff not included]

Advanced Features
*****************

What you've seen so far is all you need for basic unit testing.  The features described in this section are additions to Check that make it easier for the developer to write, run, and analyse tests.

Running Multiple Cases
======================

What happens if we pass -1 as the amount in money_create()?  What should happen?  Let's write a unit test. Since we are now testing limits, we should also test what happens when we create Money where amount == 0.  Let's put these in a separate test case called ``Limits'' so that money_suite is changed like so:

          [file check_money.3-6.c.diff not included]

Now we can rerun our suite, and fix the problem(s). Note that errors in the ``Core'' test case will be reported as ``Core'', and errors in the ``Limits'' test case will be reported as ``Limits'', giving you additional information about where things broke.

          [file money.5-6.c.diff not included]

No Fork Mode
============

Check normally forks to create a separate address space.  This allows a signal or early exit to be caught and reported, rather than taking down the entire test program, and is normally very useful.  However, when you are trying to debug why the segmentation fault or other program error occurred, forking makes it difficult to use debugging tools.  To define fork mode for an SRunner object, you can do one of the following:

 1. Define the CK_FORK environment variable to equal ``no''.

 2. Explicitly define the fork status through the use of the following function:

     void srunner_set_fork_status (SRunner * sr, enum fork_status fstat);

The enum fork_status allows the fstat parameter to assume the following values: CK_FORK and CK_NOFORK.  An explicit call to srunner_set_fork_status() overrides the CK_FORK environment variable.

Test Fixtures
=============

We may want multiple tests that all use the same Money.  In such cases, rather than setting up and tearing down objects for each unit test, it may be convenient to add some setup that is constant across all the tests in a test case.  In the Extreme Programming (XP) approach to unit tests, each such setup is called a test fixture.

A fixture is created by defining a setup and/or a teardown function, and associating it with a test case.  There are two kinds of test fixtures in Check: checked and unchecked fixtures. These are defined as follows:

Checked fixtures
     are run inside the address space created by the fork to create the unit test.  Before each unit test in a test case, the setup() function is run, if defined. After each unit test, the teardown() function is run, if defined.  Since they run inside the forked address space, if checked fixtures signal or otherwise fail, they will be caught and reported by the SRunner.

Unchecked fixtures
     are run in the same address space as the test program. Therefore they may not signal or exit, but may use the fail functions. The unchecked setup(), if defined, is run before the test case is started. The unchecked teardown(), if defined, is run after the test case is done.

Test Fixture Examples
---------------------

We create a test fixture in Check as follows:

 1. Define global variables, and functions to setup and teardown the globals.  The functions both take void and return void. In our example, we'll make five_dollars be a global created and freed by setup() and teardown() respectively.

 2. Add the setup() and teardown() functions to the test case with tcase_add_checked_fixture().  In our example, this belongs in the suite setup function money_suite.

 3. Rewrite tests to use the globals.  We'll rewrite our first to use five_dollars.

Note that the functions used for setup and teardown do not need to be named setup() and teardown(), but they must take void and return void.  We'll update check_money.c as follows:

          [file check_money.6-7.c.diff not included]

Checked vs Unchecked Fixtures
-----------------------------

Checked fixtures run once for each unit test in a test case, and so they should not be used for expensive setup.  On the other hand, since unchecked fixtures may take down the entire test program, they should only be used if they are known to be safe.  Recall that unchecked fixtures run once per test case, as opposed to once per unit test.

Additionally, checked and unchecked fixtures may behave differently in CK_NOFORK mode:

  * Normally, in CK_FORK mode, unit tests may abuse the objects created in an unchecked fixture with impunity, without affecting other unit tests in the same test case, because the fork creates a separate address space.

  * However, in CK_NOFORK mode, all tests live in the same address space, and side effects in one test will affect the unchecked fixture for the other tests.

A checked fixture will generally not be affected by unit test side effects, since the setup() is run before each unit test.  There is an exception for side effects to the total environment in which the test program lives: for example, if the setup() function initializes a file that a unit test then changes, the combination of the teardown() function and setup fuction must be able to restore the environment for the next unit test.

If the setup function in a fixture fails, in either checked or unchecked fixtures, the unit tests for the test case, and the teardown function for the fixture will not be run.  A fixture error will be created and reported to the SRunner.

Multiple Suites in one SRunner
==============================

In a large program, it will be convenient to create multiple suites, each testing a module of the program.  While one can create several test programs, each running one Suite, it may be convenient to create one main test program, and use it to run multiple suites.  The Check test suite provides an example of how to do this. The main testing program is called check_check, and has a header file that declares suite creation functions for all the module tests:

     Suite *make_sub_suite (void);
     Suite *make_sub2_suite (void);
     Suite *make_master_suite (void);
     Suite *make_list_suite (void);
     Suite *make_msg_suite (void);
     Suite *make_log_suite (void);
     Suite *make_limit_suite (void);
     Suite *make_fork_suite (void);
     Suite *make_fixture_suite (void);
     Suite *make_pack_suite (void);

The function srunner_add_suite() is used to add additional suites to an SRunner.  Here is the code that sets up and runs the SRunner in the main() function in check_check_main.c:

     SRunner *sr;
     sr = srunner_create (make_master_suite ());
     srunner_add_suite (sr, make_list_suite ());
     srunner_add_suite (sr, make_msg_suite ());
     srunner_add_suite (sr, make_log_suite ());
     srunner_add_suite (sr, make_limit_suite ());
     srunner_add_suite (sr, make_fork_suite ());
     srunner_add_suite (sr, make_fixture_suite ());
     srunner_add_suite (sr, make_pack_suite ());

Testing Signal Handling
=======================

To enable testing of signal handling, there is a function tcase_add_test_raise_signal() which is used instead of tcase_add_test().  This function takes an additional signal argument, specifying a signal that the test expects to receive.  If no signal is received this is logged as a failure.  If a different signal is received this is logged as an error.

The signal handling functionality only works in CK_FORK mode.

Looping Tests
=============

Looping tests are tests that are called with a new context for each loop iteration.  This makes them ideal for table based tests.  If loops are used inside ordinary tests to test multiple values, only the first error will be shown before the test exits.  However, looping tests allow for all errors to be shown at once, which can help out with debugging.

Adding a normal test with tcase_add_loop_test() instead of tcase_add_test() will make the test function the body of a for loop, with the addition of a fork before each call.  The loop variable _i is available for use inside the test function; for example, it could serve as an index into a table.  For failures, the iteration which caused the failure is available in error messages and logs.

Start and end values for the loop are supplied when adding the test. The values are used as in a normal for loop.  Below is some pseudo-code to show the concept:

     for (_i = tfun->loop_start; _i < tfun->loop_end; _i++)
     {
       fork();      /* New context */
       tfun->f(_i);  /* Call test function */
       wait();      /* Wait for child to terminate */
     }

An example of looping test usage follows:

     static const int primes[5] = {2,3,5,7,11};

     START_TEST (check_is_prime)
     {
       fail_unless (is_prime(_i));
     }
     END_TEST

     ...

     tcase_add_loop_test (tcase, check_is_prime, 0, 5);

Looping tests work in CK_NOFORK mode as well, but without the forking.  This means that only the first error will be shown.

Test Timeouts
=============

To be certain that a test won't hang indefinitely, all tests are run with a timeout, the default being 3 seconds.  If the test is not finished within that time, it is killed and logged as an error.

The timeout for a specific test case, which may contain multiple unit tests, can be changed with the tcase_set_timeout() function. The default timeout used for all test cases can be changed with the environment variable CK_DEFAULT_TIMEOUT, but this will not override an explicitly set timeout.  All timeout arguments are in seconds and a timeout of 0 seconds turns off the timeout functionality.

Test timeouts are only available in CK_FORK mode.

Determining Test Coverage
=========================

The term code coverage refers to the extent that the statements of a program are executed during a run.  Thus, test coverage refers to code coverage when executing unit tests.  This information can help you to do two things:

  * Write better tests that more fully exercise your code, thereby improving confidence in it.

  * Detect dead code that could be factored away.

Check itself does not provide any means to determine this test coverage; rather, this is the job of the compiler and its related tools.  In the case of gcc this information is easy to obtain, and other compilers should provide similar facilities.

Using gcc, first enable test coverage profiling when building your source by specifying the -fprofile-arcs and -ftest-coverage switches:

     $ gcc -g -Wall -fprofile-arcs -ftest-coverage -o foo foo.c foo_check.c

You will see that an additional .gcno file is created for each .c input file.  After running your tests the normal way, a .gcda file is created for each .gcno file.  These contain the coverage data in a raw format.  To combine this information and a source file into a more readable format you can use the gcov utility:

     $ gcov foo.c

This will produce the file foo.c.gcov which looks like this:

          -:   41:     * object */
         18:   42:    if (ht->table[p] != NULL) {
          -:   43:        /* replaces the current entry */
      #####:   44:        ht->count--;
      #####:   45:        ht->size -= ht->table[p]->size +
      #####:   46:          sizeof(struct hashtable_entry);

As you can see this is an annotated source file with three columns: usage information, line numbers, and the original source.  The usage information in the first column can either be '-', which means that this line does not contain code that could be executed; '#####', which means this line was never executed altough it does contain code---these are the lines that are probably most interesting for you; or a number, which indicates how often that line was executed.

This is of course only a very brief overview, but it should illustrate how determining test coverage generally works, and how it can help you.  For more information or help with other compilers, please refer to the relevant manuals.

Test Logging
============

Check supports an operation to log the results of a test run.  To use test logging, call the srunner_set_log() function with the name of the log file you wish to create:

     SRunner *sr;
     sr = srunner_create (make_s1_suite ());
     srunner_add_suite (sr, make_s2_suite ());
     srunner_set_log (sr, "test.log");
     srunner_run_all (sr, CK_NORMAL);

In this example, Check will write the results of the run to test.log.  The print_mode argument to srunner_run_all() is ignored during test logging; the log will contain a result entry, organized by suite, for every test run.  Here is an example of test log output:

     Running suite S1
     ex_log_output.c:8:P:Core:test_pass: Test passed
     ex_log_output.c:14:F:Core:test_fail: Failure
     ex_log_output.c:18:E:Core:test_exit: (after this point) Early exit
     with return value 1
     Running suite S2
     ex_log_output.c:26:P:Core:test_pass2: Test passed
     Results for all suites run:
     50%: Checks: 4, Failures: 1, Errors: 1

XML Logging
-----------

The log can also be written in XML.  The following functions define the interface for XML logs:

     void srunner_set_xml (SRunner *sr, const char *fname);
     int srunner_has_xml (SRunner *sr);
     const char *srunner_xml_fname (SRunner *sr);

The only thing you need to do to get XML output is call srunner_set_xml() before the tests are run.  Here is an example of the same log output as before but in XML:

     <?xml version="1.0"?>
     <testsuites xmlns="http://check.sourceforge.net/ns">
       <datetime>2004-08-20 12:53:32</datetime>
       <suite>
         <title>S1</title>
         <test result="success">
           <path>.</path>
           <fn>ex_xml_output.c:8</fn>
           <id>test_pass</id>
           <description>Core</description>
           <message>Passed</message>
         </test>
         <test result="failure">
           <path>.</path>
           <fn>ex_xml_output.c:14</fn>
           <id>test_fail</id>
           <description>Core</description>
           <message>Failure</message>
         </test>
         <test result="error">
           <path>.</path>
           <fn>ex_xml_output.c:18</fn>
           <id>test_exit</id>
           <description>Core</description>
           <message>Early exit with return value 1</message>
         </test>
       </suite>
       <suite>
         <title>S2</title>
         <test result="success">
           <path>.</path>
           <fn>ex_xml_output.c:26</fn>
           <id>test_pass2</id>
           <description>Core</description>
           <message>Passed</message>
         </test>
       </suite>
       <duration>0.304875</duration>
     </testsuites>

Conclusion and References
*************************

The tutorial and description of advanced features has provided an introduction to all of the functionality available in Check. Hopefully, this is enough to get you started writing unit tests with Check.  All the rest is simply application of what has been learned so far with repeated application of the ``test a little, code a little'' strategy.

For further reference, see Kent Beck, ``Test-Driven Development: By Example'', 1st ed., Addison-Wesley, 2003.  ISBN 0-321-14653-0.

If you know of other authoritative references to unit testing and test-driven development, please send us a patch to this manual.

AM_PATH_CHECK
*************

The AM_PATH_CHECK() macro is defined in the file check.m4 which is installed by Check.  It has some optional parameters that you might find useful in your configure.ac:
AM_PATH_CHECK([MINIMUM-VERSION,
               [ACTION-IF-FOUND[,ACTION-IF-NOT-FOUND]]])

AM_PATH_CHECK does several things:

 1. It ensures check.h is available

 2. It ensures a compatible version of Check is installed

 3. It sets CHECK_CFLAGS and CHECK_LIBS for use by Automake.

If you include AM_PATH_CHECK() in configure.ac and subsequently see warnings when attempting to create configure, it probably means one of the following things:

 1. You forgot to call aclocal.  autoreconf will do this for you.

 2. aclocal can't find  check.m4.  Here are some possible solutions:

      a. Call aclocal with -I set to the location of check.m4.  This means you have to call both aclocal and autoreconf.

      b. Add the location of check.m4 to the dirlist used by aclocal and then call autoreconf.  This means you need permission to modify the dirlist.

      c. Set ACLOCAL_AMFLAGS in your top-level Makefile.am to include -I DIR with DIR being the location of check.m4.  Then call autoreconf.

Copying This Manual
*******************

     [file fdl.texi not included]