[Axiom-developer] Re: [sage-devel] test suites, was: presentation about Maxima at Sage developer days

[root]

>> Are there any other major test suite collections available?
>
>At present Maxima includes a copy of Michael Wester's test suite
>which was the basis for his published review of computer algebra
>systems from about 10 years ago. We haven't done anything with it
>but I asked for and received permission from him to release it under
>terms of GPL and it is now in Maxima CVS.

Yes, I did a set of tests for Michael years ago for the review.
The Axiom version of those tests are in our test suite also.

I'm looking for test suites against "standard" sources (like Schaums,
Zwillinger, Kamke, etc) as well as other large collections of tests
(e.g. Bondarenko). Ideally ALL computer algebra systems should
test against the same published standard sets, a project I call "CATS"
(Computer Algebra Test Suite).

It only makes sense to have a fully vetted version of tests that
people can use against all the systems. These tests should have the
agreed-upon "correct" answers along with commentary about ranges,
branch cuts, etc. Ideally these CATS documents would also present the
algorithms in both mathematical terms and as software implementations.
(I would support Sage's calc rewrites IF they were doing it to present
the algorithms in some literate form for people to reference and
learn. Sadly competition, not science, seems to be the driving
motivation.)

If we can collect such test sets and show that each system generates
equivalent answers (a difficult problem given the zero equivalence
issue), then people can feel SOME confidence that 
  (a) they are getting reasonable answers
  (b) the systems are plug-equivalent

This IS supposed to be computational mathematics and the answers
are testable.

In fact, I believe that NIST should fund such an effort to support
both the quality of computational mathematics and the educational
aspects of showing the "best of breed" computational algorithms.

Tim