5.7. Writing good error messages

The message that is printed when a test fails is extremely important.

Someone will take your test, run it on a Windows platform that you don't have access to, and discover that it fails. They will then post an email with the output of the test, and in particular your error message. Someone, maybe you, will then have to figure out from this error message why the test failed.

If the error message contains all the relevant information that will be easy. If not, then it will require modifying the test, finding someone to compile it on Windows, sending the modified version to the original tester and waiting for his reply. In other words, it will be long and painful.

So how do you write a good error message? Let's start with an example of a bad error message:

    ok(GetThreadPriorityBoost(curthread,&disabled)!=0,
       "GetThreadPriorityBoost Failed\n");

This will yield:

thread.c:123: Test failed: GetThreadPriorityBoost Failed

Did you notice how the error message provides no information about why the test failed? We already know from the line number exactly which test failed. In fact the error message gives strictly no information that cannot already be obtained by reading the code. In other words it provides no more information than an empty string!

Let's look at how to rewrite it:

    BOOL rc;
...
    rc=GetThreadPriorityBoost(curthread,&disabled);
    ok(rc!=0 && disabled==0,"rc=%d error=%ld disabled=%d\n",
       rc,GetLastError(),disabled);

This will yield:

thread.c:123: Test failed: rc=0 error=120 disabled=0

When receiving such a message, one would check the source, see that it's a call to GetThreadPriorityBoost, that the test failed not because the API returned the wrong value, but because it returned an error code. Furthermore we see that GetLastError() returned 120 which winerror.h defines as ERROR_CALL_NOT_IMPLEMENTED. So the source of the problem is obvious: this Windows platform (here Windows 98) does not support this API and thus the test must be modified to detect such a condition and skip the test.

So a good error message should provide all the information which cannot be obtained by reading the source, typically the function return value, error codes, and any function output parameter. Even if more information is needed to fully understand a problem, systematically providing the above is easy and will help cut down the number of iterations required to get to a resolution.

It may also be a good idea to dump items that may be hard to retrieve from the source, like the expected value in a test if it is the result of an earlier computation, or comes from a large array of test values (e.g. index 112 of _pTestStrA in vartest.c). In that respect, for some tests you may want to define a macro such as the following:

#define eq(received, expected, label, type) \
        ok((received) == (expected), "%s: got " type " instead of " type "\n", (label),(received),(expected))

...

    eq( b, curr_val, "SPI_{GET,SET}BEEP", "%d" );