Defining Input for Regression Testing: The Test Facility

When developing a grammar, it is important to check that existing input configurations remain compatible whenever the grammar is changed. To this end, FUF includes a regression testing mechanism an a test system. This system allows the grammar developer to define a set of test cases, which are pairs of input FDs together with the sentence they are expected to produce. The test system allows to run a series of tests and checks whether the output is equal to the expected output, it can in addition measure the time spent on each test. A final advantage of the test facility is that test definitions can use the :& notation and therefore big FD specifications can be written modularly.  

The following functions are defined to take advantage of this facility:

          
(def-test name result input)
;; Define a named test: test on input should produce result.
;; If result is a list, testing result can produce any one of the elements
;; of result.


(get-test name)
;; Return the input for named test.


(clear-tests)
;; Remove all test definitions.


(test &key from to item timed)
;; Evaluate a sequence of tests.
;; A test calls uni-string on input and compares the result with the test's
;; result.
;; If timed is non-nil, time all tests.
;; from and to identify first and last tests in order in which they have
;; been defined.
;; item: identify tests explicitly - either one name or a list of names.


Examples:
(def-test :from 't1 :to 't212)
(def-test :item '(t6 t9 t45) :timed t)
(def-test :item 't6)
          
          

When running the tests, the following output is produced:

          
FUG5 12> (test :item '(t1 t14 t212))


====================
T1 -> `This car is expensive.'


[Used 117 backtracking points - 26 wrong branches - 15 undos]


[Used 119 backtracking points - 26 wrong branches - 15 undos]
OK
====================


T14 -> `For her to do it.'


[Used 152 backtracking points - 36 wrong branches - 13 undos]


[Used 158 backtracking points - 36 wrong branches - 13 undos]
OK
====================


T212 -> `What.'


[Used 39 backtracking points - 12 wrong branches - 4 undos]


[Used 40 backtracking points - 12 wrong branches - 4 undos]
OK
====================


3 tests run - 3 correct.
          
          

If a test does not succeed, the following output is produced:

          
FUG5 14> (test :item 't1bis)


====================
T1BIS -> `This car is not expensive.'


[Used 117 backtracking points - 26 wrong branches - 15 undos]


[Used 119 backtracking points - 26 wrong branches - 15 undos]
Expected `This car is not expensive.'
Instead  `This car is expensive.'
====================


1 test run - 0 correct.
The following tests are incorrect: (T1BIS)