During a recent SVG Working group discussion, we talked about how to improve the SVG test suite. Part of the problems in the development of the test suite is the sheer number of tests to be produced (i.e. written + reviewed + approved) and then tested against every implementation. The purpose of the testing is to check if every feature passes in at least two independent implementation: that is the criteria to exit Candidate Recommendation. The SVG Tiny 1.2 test suite already contains 496 tests, the SVG 1.1 2nd edition has 526. The SVG 2 spec will probably have more. Having one person test all features is not realistic anymore.

The CSS WG has been working for a long time on this issue. CSS 2.1 has 9685 tests! Together with a team from W3C, the WG set up a test suite server, where one can browse the CSS Test Suite, test its viewer and report the results. All results are then aggregated. The W3C architecture allows for 2 types of tests:

• ref tests (see Mozilla’s use of ref tests), where a test is provided together with an alternative way of producing the same result (visually).
• JavaScript tests, where together with the test, one provides JavaScript calls to test that the DOM tree is in the expected state.

Both of these types require quite some work to produce either the reference image or the expected results. This can be tedious. Also, for interactive cases, one would have to create the script to test all interaction paths. And for animation, one would have to author the JavaScript to test changes at some point in time.

In GPAC, we experimented a different approach. Continue reading Automated Testing Script Generation →