# Monday, April 23, 2007
« Detecting .NET 2.0 x64 | Main | JCK Certification »
Running Application Test Suites

Last week an anonymous person filed three bugs (1701353, 1701738 and 1701756). The first one was a URI bug that was already fixed in GNU Classpath, but the other two were more interesting because they included references to Eclipse projects that include JUnit test suites that reported some failures.

Now this is not always the case, but running the test suites for these two projects was a joy and resulted in several bugs filed/fixed:

Random MTJ test failures GNU Classpath java.util.Arrays.sort() bug
JRuby test_bignum failure GNU Classpath java.math.BigInteger.mod() bug
JRuby test_zlib failure GNU Classpath java.util.zip.GZIPInputStream constructor bug
Random JRuby test_thread_group hang        IKVM.NET thread termination deadlock bug

The reason that the MTJ tests failed in a random way was because the tests are non-deterministic. They generate random sized matrices, so the sort bug wouldn't always result in an incorrect sorting of the array indices.

Unfortunately I had to close 1701738 as Wont Fix. The JRuby test_array test intentionally causes a stack overflow and then JRuby expects to recover from that by catch StackOverflowError, but it's not possible for IKVM to reliably support catching a System.StackOverflowException and then mapping it to java.lang.StackOverflowError. The mapping exists, but if the exception is caught while there is not enough stack space for the mapping code to run, the CLR terminates the application.

The are two more JRuby tests that fail: test_io and test_pipe. These tests fail due to the fact that I haven't yet implemented NIO pipes. This is the first time ever that I've seen code "use" NIO pipes, but since it's only a test case I'm still not very motivated to implement them (but patches are welcome, as are applications that use NIO pipes).

If anyone has any other suggestions for test suites (that are easy to run like these), I'd love to hear them.

Monday, April 23, 2007 7:42:14 AM (W. Europe Daylight Time, UTC+02:00)  #    Comments [5]
Monday, April 23, 2007 8:03:48 AM (W. Europe Daylight Time, UTC+02:00)
Thats some serious comment spam thwarter - I had to look up the answer! Might I suggest Askimet? It rocks.

Anyway, was wondering, do you think IKVM could ever become a certified java run time? (provided Sun made the JCK available under a reasonable license)
Monday, April 23, 2007 10:17:01 AM (W. Europe Daylight Time, UTC+02:00)
On Gentoo we support running the application test suites when installing stuff so you have access to a vast amount of test suites for Java stuff.
Monday, April 23, 2007 12:50:50 PM (W. Europe Daylight Time, UTC+02:00)
Now this is cool...IKVM running JRuby running Ruby tests to find bugs in IKVM. Hot damn!

That's too bad about the stack thing...is that a CLR problem? The way we're able to construct a Ruby stack error on Java stack overflow is to catch it higher up, where it could be useful. Basically, at the typical places where such an exception could be expected to pass through, we re-wrap. We almost never run into a case where the stack exception happens at exactly that point...and even if it did, we'd just propagate to the next highest level. Perhaps something similar would work for you?

Also, I'm not sure if you were intending to run JRuby or advocate running JRuby, but I'd love to hear more about it. And you're going to find that test_io and test_pipe are fairly important tests for Ruby apps...
Thursday, April 26, 2007 10:14:13 PM (W. Europe Daylight Time, UTC+02:00)
I was that anonymous person (too lazy to log into SourceForge). Glad to see that these third party libraries turned up some IKVM/Classpath issues. I'd definately give some more Java test suites a go in the near future. *.apache.org anyone? :-)

Hopefully something can be done about the JRuby corner cases. JRuby.NET would just be too sweet. :-)
Monday, April 30, 2007 1:01:32 AM (W. Europe Daylight Time, UTC+02:00)
1) javolution (http://javolution.org/) has a tiny test suit but a very large benchmark suit. It can be built for Java 1.4 or 1.5 or even specifically target GCJ (configurable) and then executing the benchmark is a matter of:
ikvm -jar javolution.jar perf
under Java 1.4 there are failures.
I think Javolution can be a strong test of JVM+runtime compatibility.

2) XStream is an object-to-XML and vice-versa serializer (no schema or bindings involved). We use it for moving objects between Websphere Application Server 6.0/6.1 and IKVM on .Net 1.1 (don't ask :) ). Even though using serialization between different JVM's is not smart, GNU Classpath is surprisingly compatible to IBM/Sun. I seem to remember a few unit tests fail when run under IKVM, though.
Nadav Wiener
Comments are closed.