CTB – test unitaire pour C

[English][Deutsch]

CTB - C Test Bed System

Information in this document corresponds to the CTB version 4.2.

1. INTRODUCTION

CTB System (C Test Bed System, or just CTB) is a state-of-the-art testing tool for the C programming language. With CTB you can very easily generate a testing environment, called test bed, for your C modules or APIs, which you wish to test. CTB supports so called specification-based (black-box) testing approach.

CTB provides you the facilities which are needed in a professional module/subsystem/API testing of C code:

  • Generation of test harness
    • Test driver
    • Stubs (if needed)
    • Makefile-based, incremental building of the test beds: important when CTB is being used in the large
  • Early module testing
    • As soon as the module compiles the test bed can be generated on it without any additional programming from your part
    • If needed, you can use stubs to emulate the functions called from other modules
  • Powerful testing environment
    • C-like command interface
    • Early tests/experiments with interactive interface
    • Use of stubs, you can freely determine the stub behavior at test bed time
    • Script file based automated tests
    • You can define expected results in advance and make the test bed to check them for you automatically
    • You can modify and extend the test suite simply by editing the script files, the time-consuming and error prone full edit-compile-link cycle is not needed
    • You can model your test scripts to testable items, test cases
    • Documentation of the test session produced automatically
      • Trace file: detailed log of what happened
      • Summary file: a compact higher abstraction level document, in terms of test cases with PASS/FAIL etc. data
    • You can use any code coverage tool in the test bed environment thus unifying the black-box testing approach of CTB and the white-box testing approach of the coverage tool
      • For example Testwell's CTC++ can be used
    • Test beds can be run under a debugger for finding the detailed reasons on the found bugs
  • Ease of use
    • Extensive on-line help in all phases, man pages (in Unix)
  • Standardized module testing practices and environments

2. HOW CTB WORKS

2.1. CTB Generator

The actual tool component of the CTB System is the CTB Generator. It analyzes the C source code based on the .h or .c files to find the interfaces to be tested and generates a C module called test driver. If you need to generate stubs, you just give the interface (.h) files of the stub modules to the generator.

The generated test driver code is compiled and linked together with the code under test resulting in a test bed program. CTB Library (test bed run time support library, part of the CTB System) is linked also into the test bed.

The rules for building the test bed can be put into a makefile. Additionally, the test beds can be built incrementally, that is, the test harness code of changed modules can be replaced and the harness code of new modules can be added into a test bed without the need to regenerate the harness code of the unchanged modules. The integrity of the generated test harness code and the code under test is taken care by the makefile. These are valuable properties when the test bed consists of very large and/or many C modules.
 

CTB test bed generation
Figure. Running the CTB Generator and compiling/linking the test bed.

2.2. CTB Test Beds

Testing is carried out with the resultant test bed program. The test driver part of it is actually a powerful C interpreter. It is partly generated code and partly fixed code from the test bed library.

The test bed user interface uses C-like commands. The commands are either read from the terminal (interactive mode) or from a script file (script mode). By default the test bed starts in interactive mode but it can also be started directly to script mode (say, form a makefile) thus requiring no user interaction for running the tests.

You can also embed the test bed, and its invocation with some set of test scripts to execute, into a C program, where you have your own main().

The look and feel of the test bed execution is as if the test driver would read the test main program and immediately execute it command by command and show what happens. The test driver knows the basics of C, the entities that there are in the interface of the code under test. There are also some additional commands for controlling the test environment (trace file, test case concept, stub behavior definition, etc.).

The test bed architecture is illustrated below.
CTB test bed running
Figure. Executing test runs with the test bed.

The basic differences to the manually coded test main programs and CTB generated test drivers is that in the former the test logic (what functions to call, in what order, with what parameter values, etc.) is hard coded in the test main program. And when such test program is executed there comes no documentation of what happens. Of course, you might have added some trace outputs into the test main program, but there you have the big question: how much effort and additional code you should put into the test main program to get the test session, visible, automatically repeatable, self-checking, measurable, standardized, etc. Changes to the manually developed hard-coded test driver require an edit-compile-link cycle.

In the CTB generated test beds the testing infrastructure (visibility, etc.) is automatically generated and similar in each test bed. Of course, also with CTB test beds you have to provide the intellectually challenging thing, the test logic (what functions to call, in what order, etc.). You can give it interactively (when quick experiments) but normally in script files. You can add new tests just by typing them interactively or editing them into the script file, the time-consuming full edit-compile-link cycle is not needed.

Some of the test bed features are described below in more detail.

2.2.1. Test Bed Variables

The main purpose of the test bed is to call the functions to be tested. Usually the functions have parameters and return values. So called test bed variables can be allocated as needed for these purposes. Test bed variables behave like true C variables although they are created dynamically during the test session: 
    BED> char c = 'x', char *d, e[20];
    BED> int i, ia[4];
    BED> my_struct r = {100, "Hello", 7.2};
Values can be assigned to test bed variables as in C: 
    BED> e[0] = 'C';
    BED> e[1] = e[0];
    BED> e[2] = '\0';
    BED> i = 2 * 10 + (1 << 3) * 6 - 3;
    BED> ia = {11, 22, 33, 44}; // well, aggregate support is beyond C...
The value of a test bed variable, as well as any expression, can be examined and the test bed can be set to display the values of evaluated expressions automatically: 
    BED> ? i;
         65
    BED> ? e;
         "CC"
    BED> SET $AUTODISPLAY ON;
    BED> i++;
         65
Values of structured variables, data behind pointers, etc. can also be displayed.

2.2.2. Calling the Functions

Calling the functions of the modules under test is, of course, the most important use of the test bed. The function call syntax is, again, same as in C. Test bed variables and external variables declared in the code under test can be used as parameters and they can receive function return values. Also constant parameters can be used. 
    BED> d = strcpy(e, "C T B");
         00DF:2520
    BED> ? d, "%s";
         "C T B";
    BED> i = strcmp(d, e);
         0

2.2.3. Test Scripts

The test beds can be run either interactively or through a test script. Scripts are text files containing the test calls expressed in the test bed command language. For example, the test script pipes.tst is executed as follows: 
    BED> execute pipes.tst;

    line 1 in file pipes.tst
    BED: PIPE_HANDLE ph;
    BED: ph = create_pipe(0x30, MAX_SIZE);
    BED: write_pipe(ph, "This is a test");
    ...
Because the test calls are expressed in terms of the module specification, they can be designed and stored in test scripts as early as at the module design phase, even if the implementation of the module is not yet available. After the test scripts have been written they can be automatically executed (and updated, if needed) with minimal effort any time the implementation of the module changes.

Script files can execute other script files. There are various options for controlling the script execution. You can visit interactive mode in the middle of a script execution and define actions to be taken on syntax errors and assert failures.

You can also start the test bed from operating system command line or from another C program and provide the script file(s) as parameter, which the test bed executes automatically.

2.2.4. Trace Listings

The test bed writes a trace listing file, which contains the commands executed during the test session and their output. The trace listing describes completely what was tested, when, how, and to which extent. Regression testing can be based on trace listings comparison.

2.2.5. Assertions

Assertions are logical expressions which are used for checking whether a condition is currently satisfied: 
    BED: status = pipe_read(ph, buf, 1000, &num_read);
    BED: assert (status == 0 && num_read == 5);
         *** CTB: Assertion failed in pipes.tst at 114
Assertions can be used as means to make the test bed to check that the actual result of a test call is as expected. If an assertion fails a message is displayed (to screen and to trace file) and current test case goes to FAIL state. Listing of the locations of the failed assertions can be obtained.

2.2.6. Test cases

Test cases are normally used in scripts. A test case is any sequence of test bed commands (test calls, assert commands and others), which make up a logical entity for testing the expected behavior of some feature. Test case is started with command 
    BED: set $testcase "name of this test case" on;
Initially the test case status is PASS. Assert violations change the state to FAIL and possible test bed errors to ERROR state. Test case is ended when a new test case begins, or explicitly with command 
    BED: set $testcase off;
When a test case ends, small test case specific writing to a summary file is done. Actually the whole rationale for test cases is that they are means for modeling the test suite (one or more script files) into logical entities, test cases, and getting the summary file, which is a compact PASS/FAIL report of the test cases.

2.2.7. Summary file

Summary file is a compact, tabular, higher abstraction level (than the detailed trace file) document of the results of the test session. The information is in terms of test cases executed.

Summary file has either a default layout or a user defined layout. An excerpt of a summary file (with default layout) is below. 

... listing header containing some useful version tracking etc.
... information, omitted here

Testcase                                         Calls   Asserts PASS/FAIL
------------------------------------------------ ------- ------- ---------
Tests on empty buffer                            4       2       PASS     
Tests on normal size buffer, no paging needed    10      5       PASS     
Tests on normal size buffer, paging needed       16      4       FAIL     
Tests when buffer is almost full                 12      12      PASS
... more test case specific outputs

Summary statistics:

       170 function call(s)
        66 evaluated assertion(s)
         8 failed assertion(s)
         0 test bed error(s)
         0 errno violation(s)
         0 heap error(s)
        13 formal test case(s) (11 PASS, 2 FAIL, 0 ERROR)

       *** Overall:  F A I L
The FAILed test cases can be located and analyzed from the detailed trace file.

The layout of the summary file can be customized with a simple test bed command. With it you can define some general header texts file and what data and in what layout the test bed writes to summary file at each test case end. For example, you could get to a summary file a listing of actual parameters, actual results, expected results and PASS/FAIL info of the test cases.

2.2.8. Control Structures

Scripts can be made to behave conditionally as the testing needs require. if, else and while control structures are supported. An example: 
     p = list_first(the_list);
     if (p == NULL)
     {
         ECHO "Oops, p is NULL\n";
         stop;
     }
     while (p != NULL)
     {
         ? p->value, "%s";
         p = list_next(the_list);
     }
A special for loop structure is supported. It is a convenient way to express large test data sets. An example below shows the idea: 
    int i, j, k;
    for    i,    j,      k   in
           1,    2,      3,
          -1,   -2,     -3,
       32767,    1, -32768,
           0,    0,      0,
    ...
    {
       set $testcase "special addition" on;
       assert(special_add(i, j) == k);
    }

2.2.9. Stubs

You can use stubs when you wish to simulate the behavior of the functions called from the code under test. Possible needs are for example:
  • Simulation of target hardware dependent features, which are not available in the testing environment (esp. doing module tests in the host machine's environment).
  • Simulation of the modules, which are not yet coded (esp. in multi-person projects, or when practicing a top-down module testing order).
  • Arranging hard to produce error conditions from the called modules (esp. when striving to high code coverage in the modules under test).
  • When the testing process rules require "testing in isolation".
  • When you need to focus your tests on some parts of the system and emulate the behavior of some other parts of the system but sometimes allow the real implementation to execute. In the same testing environment you could have many test suites, each having their own test focus and set of subsystem, whose behavior is emulated during the given test suite.
Basic use of the stubs:

When building the test bed you just show to the CTB generator the interface (.h) file of the module on whose extern functions you wish to have stubs generated. Then, at test bed time you can determine what each of the stub functions will do, when they eventually get called. For example, as follows: 

    BED> set stub foo
    BED+ {
    BED+     ?par1;
    BED+     assert (par1 < 100);
    BED+     *par2 = '\0';
    BED+     return 1;
    BED+ }
Above par1 and par2 are assumed to be formal parameters of the foo function and you can access them like test bed variables. In overall, you in a way give a small script, which will be executed each time the stub function is called.

By default stub execution is traced in the same way as any other test bed commands. In heavy testing you can reduce the possible excessive trace output from stubs by means of stub verbosity, for example: 

    BED> set stub foo verbose 3 then brief;
    BED> set stub bar silent;
Now the stub foo will give full trace of its behavior on its three first calls (you'll see that it got called and how the few first calls went), thereafter it only signals that it was called. The stub bar doesn't report anything (except possible assert violations) of its executions.

In some interactive test sessions you might want to manually control what will be done when a stub is called. This is possible as illustrated in the following example: 

    BED> set stub foo interactive;
    BED> some_function(some_parameters); // causes foo to be called...

    Stub function foo called
    BED# ?par2;     // this line typed from terminal
         5FD4:0008
    BED# return 1;  // this line typed from terminal

    BED> ...
Or perhaps you let the stub go automatic for some number of times and then take over with manual control... 
    BED> set stub foo active 500 then interactive;
Stub capability invoking from the user's code:

This is quite similar as the basic way of using stubs except the following:

As in the basic style you show the stub module interfaces to the CTB generator and mark these functions to be stubs but for which function implementation code will not be generated. Instead, you provide the implementation for these functions and insert one simple macro call into your code to enable the stub behavior.

When this kind of stub function gets called during testing, you can control its execution in the same way as described before (basic use of stubs). Moreover, you have the following additional features available:

You can resume from the stub script execution (or from an interactive stub execution) to the real implementation of the stub function. This arrangement gives you a nice way to trace with what parameters functions were called or make assertions on the parameter values.

At your choice you can by-pass the stub behavior altogether, i.e. only your real implementation of function is executed. An example: 

    BED> set stub foo2 active 1 // the stub script executed once (which
    BED+                        // may or may not resume the execution
    BED+                        // to the real implementation)...
    BED+     then inactive;     // ...then no stub intervention at all
Inline stubs:

These are macro based arrangements mostly due to compatibility reasons to the previous version of CTB. There are some nice special things (execution timing, checking for correct call order, etc.) that you can do with inline stubs, though.

2.2.10. On-line Help and Information Queries

An interactive on-line help covering all test bed features is available during the testing session. In Unix man pages are supported.

The tester can also query various information of the test bed session. For example the following can be queried: how many times each function was called in the session, how many assertions was evaluated, how many of them failed (and where), how many test cases was executed, how many of them PASSed or FAILed.

3. INTERFACES TO OTHER TOOLS

When an error is found by module tests, the originator of the error can be located by running the test bed under a debugging tool.

The code coverage of the tests can be measured using the test bed together with CTC++, Test Coverage Analyzer for C/C++, which is another testing tool from Testwell.

4. SUMMARY OF BENEFITS

CTB is a powerful test tool for automating the module and subsystem testing of C code.
Testing Is More Automatic
With CTB generator you can automatically generate a testing environment, a test bed, for your C modules. With the test bed using test scripts you can automatically execute the tests on your modules.
Testing Is Documented and Visible
The test scripts written by the tester documents what will be tested. Trace listing is a document describing what happened in the test session in a detailed level. Summary listing is compact PASS/FAIL report of the test cases.
Testing Is Efficient
The tester has more time for designing the test cases thus increasing the test efficiency. When the tester has a convenient and versatile testing environment he has more time and motivation for finding errors (vs. fighting with the testing environment).
Tests Are Repeatable
Tests scripts, assertions, and trace listings enable repeatable testing. Tests can be easily repeated after software modification and the test results are checked by assertions and by comparing the trace and summary listings.
Testing in Isolation, More Testing in the Host Machine
Use of stubs makes this possible.
Testing Environment Is Uniform And Reliable
When using CTB all the testers have a uniform working environment compared to hand made test beds, which tend to be error prone, often undocumented and thus difficult to use. All test cases can be expressed with C-like test bed command language and the trace and summary listings have a uniform layout.
Versatile Support for Black Box Testing
CTB supports the whole spectrum of black box testing from ad hoc trials to systematic script-based regression testing. In addition to traditional black box testing CTB is also a tool for incremental software development and for evaluating software component libraries.

5. OPERATING ENVIRONMENTS

The resource requirements of your normal C-development environment are sufficient for using the CTB. CTB System can be easily ported to new ANSI C environments.

6. USER MANUAL

The CTB User Manual is available as pdf-file (user id and password needed).

last updated: 22.07.2005

© 1998-2005 Testwell Oy
CTA++, CTB, CTC++, CMT++ and CMTJava are products of Testwell Oy, Tampere (Finland)
all other trademarks of this site are the property of their respective owners.