During a recent SVG Working group discussion, we talked about how to improve the SVG test suite. Part of the problems in the development of the test suite is the sheer number of tests to be produced (i.e. written + reviewed + approved) and then tested against every implementation. The purpose of the testing is to check if every feature passes in at least two independent implementation: that is the criteria to exit Candidate Recommendation. The SVG Tiny 1.2 test suite already contains 496 tests, the SVG 1.1 2nd edition has 526. The SVG 2 spec will probably have more. Having one person test all features is not realistic anymore.
The CSS WG has been working for a long time on this issue. CSS 2.1 has 9685 tests! Together with a team from W3C, the WG set up a test suite server, where one can browse the CSS Test Suite, test its viewer and report the results. All results are then aggregated. The W3C architecture allows for 2 types of tests:
- ref tests (see Mozilla’s use of ref tests), where a test is provided together with an alternative way of producing the same result (visually).
In GPAC, we experimented a different approach. Continue reading Automated Testing Script Generation