Re: Benchmark / Performance Test in mauve

[Mark Wielaard]

Hi,

A long time ago Robert posted the following. Since a couple of people
are now trying to get DaCapo of the ground now that it has been released
under a Free Software license I thought it nice to respond (since nobody
else did before).

On Wed, 2005-09-14 at 14:51 +0200, Robert Schuster wrote: 
> this topic may belong to mauve but I am not sure whether all the VM 
> implementors
> are on that list.
> 
> Recently there was a discussion[0] on the sdljava list about the efficieny of
> direct NIO buffers and Java arrays. It took place because OpenGL functions 
> like
> glVertex3fv(float[]) in C where mapped to Java methods like this
> glVertex3f(FloatBuffer). If you don't know OpenGL: the functions expect that 
> the
> array/buffer contains only 3 values.
> 
> A user of the sdljava library (which is currently tied together with gljava)
> found out that it is better to use Java arrays as arguments here. He expected
> his application to do frequent allocations of the buffer/arrays and this would
> kill performance in case of the direct NIO buffers.
> 
> To underly this he wrote a small benchmark (which we are allowed to license
> under GNU GPL[1])
> 
> I remember that once upon a time Michael said we should test for performance
> regressions in mauve and I think the attached benchmark could be used for 
> that.
> 
> Ideas, thoughts anything?

We should test for performance regressions, but on a bit higher level.
The benchmark you attached really is a micro benchmark which does not
always represent what real world code does. (Although in this case in
clearly demonstrated that a small array of floats is more efficient then
using a FloatBuffer in general.)

It would be nice to have a collection of micro-benchmarks, and maybe you
can add a new module to mauve for this. But that should not be used as
performance regression tester. A couple of people is now trying to get
DaCapo fully supported on all GNU Classpath runtimes (to see who has the
biggest and fastest!) now that it has been re-released under a Free
Software license. It contains of a collection of real world programs
that are a better indicator for performance regressions that matter to
our users. Check it out: http://www-ali.cs.umass.edu/DaCapo/gcbm.html
Currently we need to work around some issues that prevent compilation
and/or running on GNU Classpath based development environments (*).
But most suites work already.

There is also Ashes, which is distributed under the GPL, but I haven't
tried it out yet. http://www.sable.mcgill.ca/ashes/

Cheers,

Mark

> [0] -
> http://sourceforge.net/mailarchive/forum.php?thread_id=8148756&forum_id=43388
> 
> [1] -
> http://sourceforge.net/mailarchive/message.php?msg_id=12939689

(*) There are some com.sun references in chart and batik that need to be
taken care of. And batik needs stroked shapes and TextLayout/font
measuring. There are the following bugs in our database now:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=22938
"java.awt.BasicStroke.createStrokedShape is a silent stub"
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=24402
"TextLayout is largely unimplemented (GdkTextLayout.getCharacterLevel)"

signature.asc
Description: This is a digitally signed message part