2021/10/21

Better test failure report

I keep Gauche's test framework ref:gauche.test intentionally simple--a test evaluates a given expression and compares its result with the expected result; if they don't agree, reports it. That's all.

It doesn't have fancy knobs and dials, but it does the job. Fancy features can be written using Gauche's other features; e.g. if you need setup/teardown, you can just wrap tests with unwind-protect. I prefer this kind of explicit code to fat frameworks in which you need to track down its documents and (sometimes) implementation to know what exactly is done.

However, there has been one frustration: I can't easily change how the test failure is reported. Especially, when a test yields a large amount of results and it doesn't agree with expected one, it is hard to tell where is the difference, by looking at the entire expected and actual results.

Now I can have it. See the following test:

(test* "Beatrice"
       ;; expected
        '("What fire is in mine ears?  Can this be true?"
          "Stand I condemned for pride and scorn so much?"
          "Contempt, farewell, and maiden pride, adieu!"
          "No glory lives behind the back of such.")
       ;; actual
       "What fire is in mine ears?  Can this be true?\n\
        Stand I condemn'd for pride and scorn so much?\n\
        Contempt, farewell! and maiden pride, adieu!\n\
        No glory lives behind the back of such.\n"
        test-check-diff           ; check
        test-report-failure-diff) ; report

The expected text and the actual text have slight difference. This reports the difference in unified diff format.

ERROR: GOT diffs:
--- expected
+++ actual
@@ -1,4 +1,4 @@
 What fire is in mine ears?  Can this be true?
-Stand I condemned for pride and scorn so much?
-Contempt, farewell, and maiden pride, adieu!
+Stand I condemn'd for pride and scorn so much?
+Contempt, farewell! and maiden pride, adieu!
 No glory lives behind the back of such.

The third argument of test* is to compare the expected and actual result. If you prepare expected text in one big string, you can just use the default one; test-check-diff adds a bit of convenience by accepting a few different formats.

The fourth argument is the main addition. It accepts a report proceudre which is called when the expected result and the actual result didn't match, with three arguments, **message**, **expected-result** and **acutual-result**. The **message** argument is the first argument passed to test*.

The test-report-failure-diff uses text.diff module to display the difference of the results in diff format (ref:text.diff).

You can customize reporting as you wish. Another custom reporting we'd like to have is to show difference of tree structures.

Please refer to the manual for the details. (Before releasing 0.9.11, you can view the draft document.

Tags: 0.9.11, Testing, text.diff