Re: The Mauve unicode testcase and VM performance

[Mark Wielaard]

Hi,

On Thu, 2004-04-08 at 18:44, Stephen Crawley wrote:
> The testcase now runs clean for JDK 1.4.2 as well as Kissme/Classpath
> from CVS.

Thanks a lot for that!

> The interesting thing is that the test run 200 times faster with JDK 1.4.2
> than with Kissme.  Yes TWO HUNDRED TIMES!
> [...]
> Question: what figures do people get with other open source VMs?

There are quite some differences between the free runtime environments.
(Total) running time goes from 1.2 seconds (native gcj) to 1.5 minutes
(kissme). This is on a AMD Athlon XP 1600+ (1.4 Ghz).
Note the interesting differences between the load, test and total time.
All times are best of three runs.

(gij 3.3.3)
$ time echo gnu.testlet.java.lang.Character.unicode | gij 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:1236ms   tests:5646ms
0 of 3578944 tests failed
 
real    0m7.141s
user    0m6.687s
sys     0m0.068s

(Kissme from CVS a few weeks ago)
$ time echo gnu.testlet.java.lang.Character.unicode | 
~/src/kissme/useful_scripts/kissme gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:11434ms   tests:81436ms
0 of 3578944 tests failed
 
real    1m33.086s
user    1m30.842s
sys     0m0.499s

(Native compiled mauve with gcj 3.3.3 -O2)
$ time echo gnu.testlet.java.lang.Character.unicode | ./mauve-native -debug
Reading unicode database...
done
Benchmark : load:594ms   tests:386ms
0 of 3578944 tests failed
 
real    0m1.211s
user    0m0.906s
sys     0m0.041s

(prototype Jikes RVM 2.3.2, don't have a development build handy.)
$ time echo gnu.testlet.java.lang.Character.unicode | rvm 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:1703ms   tests:1292ms
0 of 3578944 tests failed
 
real    0m6.169s
user    0m3.637s
sys     0m0.232s

(JamVM 1.1.2 with some local patches - that shouldn't impact runtime speed.)
$ time echo gnu.testlet.java.lang.Character.unicode | jamvm 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:3927ms   tests:7478ms
0 of 3578944 tests failed
 
real    0m11.572s
user    0m10.721s
sys     0m0.447s

(SableVM 1.1.2-pre as posted by Grzegorz on Planet Classpath yesterday.)
$ time echo gnu.testlet.java.lang.Character.unicode | sablevm -Y 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:2296ms   tests:6114ms
0 of 3578944 tests failed
 
real    0m8.714s
user    0m8.186s
sys     0m0.067s

(Kaffe 1.1.4 Just-in-time v3)
$ time echo gnu.testlet.java.lang.Character.unicode | kaffe 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:1080ms   tests:618ms
0 of 3578944 tests failed
 
real    0m1.985s
user    0m1.622s
sys     0m0.067s

(IKVM.NET 1.0.1549.20346 [latest snaphot]
 with Mono JIT compiler version 0.31)
$ time echo gnu.testlet.java.lang.Character.unicode | mono ikvm.exe 
gnu.testlet.SimpleTestHarness -debug
Reading unicode database...
done
Benchmark : load:2725ms   tests:1393ms
0 of 3578944 tests failed
 
real    0m5.656s
user    0m0.075s
sys     0m0.013s

signature.asc
Description: This is a digitally signed message part