Re: Tests, Documentation

[Mark Wielaard]

Hi,

On Sat, 2003-11-22 at 18:21, Sascha Brawer wrote:
> having read quite a bit of Classpath code, I'd like to express two wishes.
> 
> First, please write more test cases. [..]
> Second, please write understandable documentation [...]
>
> In other words, I think that we really need to improve the quality of our
> code. Building a sound foundation for others is not the same as some fun
> week-end hack.

Thanks for raising these issues. Between you and Dalibor policing the
code so thoroughly I feel less and less need to play the bad evil
maintainer :)

I agree with you completely. But I feel we are going in the right
direction. Although I admit that creating more tests and documentation
is a slow process. There are a couple of things we should do to make
this easier (and more fun!):

- More peer code-review. It is really good to see that people post their
  patches to the mailing list so others can comment on it. We probably
  need a new list for it (classpath-patches) and a good set of rules of
  who and what kind of changes can be approved/denied or asked for
  improvements. I personally find peer review discussions very valuable,
  even though people often point out my mistaken assumptions.

- Always adding mauve tests for new code or patches. Mauve tests are
  extremely simple to write and give you immediate feedback. The amount
  of tests is also big enough to catch things you might not have thought
  would be changed by your patch. But as you point out there are areas
  with not very much or even no tests at all.
  A few problems with mauve currently are:
  - It is a pain to setup. We need to get rid of the auto* magic.
  - It is a bit to fragile. A hanging test or no network can stop
    everything. I have a simple shell/bash based wrapper to work around
    that.
  - There are still to many false positives.
  - Mauve proper doesn't have a good XFAILs mechanism.
    gcj does have something like that (although it doesn't detect when a
    XFAIL turns into a XPASS).
  All the above makes spotting regressions not easy (some of the
  recent network/url/connection breakage should have been spotted much
  earlier).

  With a few local patches I currently get for Classpath+Kissme (x86):
  182 of 10352 tests failed
  This includes the patch for GregorianCalendar and merging in
  URLConnection and gnu/java/net/protocol from libgcj. Some outstanding
  java.awt.geom patches and the recent Hashtable fixes that are not yet
  checked in.
  I had to disabling some tests (see attached mauve-kissme file).
  Most of the remaining failures are some char-encoding issues,
  bugs related to relection,  java.text.RuleBasedCollator (for which
  I believe we still have a patch pending) and some
  RoundRectangle2D.intersects failures. Plus a couple of smaller test
  failures.

- Autobuild Classpath, VMs, Mauve and applications regularly.
  Tom Tromey recently posted a list of free software projects that he
  regularly compiles against gcj and for which he runs the tests-suites.
  It would be good to create a classpath-testresults mailinglist and
  post the from a couple of different architectures, compilers, VMs,
  etc. Any volunteers to adapt Tom his scripts and to run this
  regularly?

- Automatic documentation generation. When updates to documentation are
  immediately visible people get much more satisfaction from writing it.
  Current problems are that gjdoc has not had an official release so you
  have to build from CVS. We need a way to add the correct copyright
  notice and distribution terms to each page (note that since we also
  have non-Classpath classes in CVS now in the external directory this
  is a bit tricky to get correct. gjdoc (at least my setup) does not
  handle all HTML entities correctly. See
  https://savannah.gnu.org/bugs/?func=detailbug&bug_id=4823&group_id=85
  And as far as I know it doesn't handle embedded images yet.

  Arnaud is currently testing gjdoc and kaffe.
  Although I hope we can fix/use gjdoc it would be nice to have a
  volunteer to check out SinjDoc from the Flex project.
  http://cscott.net/Projects/GJ/#sinjdoc
  - Does it work with gcj/kaffe?
  - Does it produce correct documentation for all of Classpath?

One additional documentation task would be to write a good book/article
on working with GNU Classpath and/or gcj/kaffe for free software
writers. This should be an overview of what is available now, how to
install it, writing hello world and more cookbook like examples with a
overview of the current packages available (when to use what for which
kind of task). Most of the articles/books about writing programs in the
java programming language and/or libraries are about proprietary systems
which make all kind of assumptions that are not appropriate when working
on a free system.

I want to do it all, but I am terribly slow :{
So volunteers for all of the above tasks are welcome!
I am currently concentrating on cleaning up Mauve and preparing the 0.07
snapshot for next week.

Cheers,

Mark

# Config file that tells mauve about the `kissme' tag.
JDK1.0
JDK1.1
JDK1.2
JDK1.3
JDK1.4
JLS1.0
JLS1.1
JLS1.2
JDBC1.0
JDBC2.0

# This test sets the JVM's SocketImpl factory to a dummy factory that
# breaks any following tests that test or use sockets.
!java.net.Socket.setSocketImplFactory

# Character.unicode seems to be very broken (the test)
# Does not give meaningfull test results at the moment.
!java.lang.Character.unicode

# These are almost certainly buggy test cases.
# The behaviour of the garbage collector cannot be predicted.
# (Note: trailing '.' is to avoid disabling the reflect testcases.)
!java.lang.ref.

# Crashes when the testcase is compiled with gcj (but not jikes 1.16 or 
# JDK 1.4.0 javac)
#!java.lang.Math.min

# Crash!
!java.lang.reflect.Array.newInstance

# Stupid security manager failures.
!java.io.File.security

# Hangs
!java.lang.Thread.join
!java.net.DatagramSocket.DatagramSocketTest

signature.asc
Description: This is a digitally signed message part