Last week an anonymous person filed three bugs (1701353, 1701738 and 1701756). The first one was a URI bug that was already fixed in GNU Classpath, but the other two were more interesting because they included references to Eclipse projects that include JUnit test suites that reported some failures.
Now this is not always the case, but running the test suites for these two projects was a joy and resulted in several bugs filed/fixed:
The reason that the MTJ tests failed in a random way was because the tests are non-deterministic. They generate random sized matrices, so the sort bug wouldn't always result in an incorrect sorting of the array indices.
Unfortunately I had to close 1701738 as Wont Fix. The JRuby test_array test intentionally causes a stack overflow and then JRuby expects to recover from that by catch StackOverflowError, but it's not possible for IKVM to reliably support catching a System.StackOverflowException and then mapping it to java.lang.StackOverflowError. The mapping exists, but if the exception is caught while there is not enough stack space for the mapping code to run, the CLR terminates the application.
The are two more JRuby tests that fail: test_io and test_pipe. These tests fail due to the fact that I haven't yet implemented NIO pipes. This is the first time ever that I've seen code "use" NIO pipes, but since it's only a test case I'm still not very motivated to implement them (but patches are welcome, as are applications that use NIO pipes).
If anyone has any other suggestions for test suites (that are easy to run like these), I'd love to hear them.