# Monday, 08 November 2004
Running Derby on IKVM

At the Colorado Software Summit I attended a presentation on Derby by Dan Debrunner. Derby is the open source version of IBM's Cloudscape database engine. After seeing the presentation I decided to try to run it on IKVM. I tried some simple tests and they all went fine. At the moment Derby doesn't include a test suite yet (IBM is working on releasing their tests), so I didn't do any extensive testing.

Since Derby is a real transactional database engine it uses a transaction log and requires FileDescriptor.sync() to work corectly, so I decided to implement that. I hadn't previously implemented it, because it unfortunately requires platform specific code, so it only works on Windows and on Posix systems (when running on Mono).

I wrote a very simple test app in C# (and the Java equivalent) and did some performance tests (download source):

Operation Sun JDK 1.5 IKVM/.NET 1.1 IKVM/Mono 1.0.2
getConnection 1903 1472 2177
create table 450 634 624
prepareStatement1 344 514 978
1000x insert 1031 387 550
prepareStatement2 124 294 231
1000x select 1038 851 3682

(Times in milliseconds, based on average of three best runs out of four runs.)

All the usual disclaimers about benchmarks apply of course, but it's interesting to see that IKVM on .NET outperforms JDK 1.5 on the queries. Preparing the statements take longs, but that is expected, as IKVM has to convert the generated class files to CIL (and this part of IKVM is not very efficient).

Another interesting aside is that when I ran the test on JDK 1.4.1, the insert operation took about 40 seconds to run. This rather extreme performance bug was fixed in 1.4.2.

One thing that is important to note when running Derby on IKVM is that Derby compiles all queries to bytecode and IKVM doesn't support garbage collecting classes (due to .NET's inability to unload code), so unless your application uses a fixed set of prepared statements that fit in the Derby statement cache, you're going to leak memory.

I made a new snapshot that includes the new sync implementation:

  • Resync'ed with GNU Classpath cvs.
  • Implemented java.io.FileDescriptor.sync() and java.nio.channels.FileChannel.force().
  • Fixed java.io.File to treat file paths as case insensitive on Windows.
  • Made exception helper a little more robust against exceptions happening early in bootstrap.
  • Fixed japi build file to include all the public APIs.
  • Fixed ClassLoaderWrapper.getSystemClassLoader() to bypass ClassLoader.getSystemClassLoader() and access the underlying field directly, to prevent a security check.
  • Changed DotNetTypeWrapper to overrider GetClassLoader to lazily get the system class loader.
  • Fixed ikvmc generated main stub to return 1 if the program exited with an exception.
  • Implemented support for java.security.manager system property (the security manager can now be set from the command line using -Djava.security.manager=<class>).

New snapshots: just the binaries and source plus binaries.

Monday, 08 November 2004 11:51:13 (W. Europe Standard Time, UTC+01:00)  #    Comments [8]
# Thursday, 04 November 2004
Japitools & New Snapshot

I've integrated Stuart Ballard's Japitools in my build process to keep track of the API completion status and to test ikvmc and ikvmstub. To results of running it on the current ikvm development snapshot can be found here.

What's new in the snapshot:

  • Implemented AWT Toolkit.getFontList().
  • Regenerated mscorlib, System and System.Xml stub jars with new version of ikvmstub.
  • Resync'ed with GNU Classpath cvs.
  • Improved error handling in Socket classes (exceptions throw now match the JDK better).
  • Implemented urgent data and shutdown methods in PlainSocketImpl (thanks to Dmitry Gromov for this).
  • Fixed ikvm.exe and ikvmc.exe to support slashes in Main-Class entry in manifest.
  • Fixed ikvmstub.exe to emit throws clauses for constructors.
  • Fixed bug that caused internal error when compiling class that has a non-private final field of a type that cannot be loaded.
  • Fixed reflection on .NET types to mark non-virtual methods as final.
  • Fixed name clash detection algorithm to ignore constructors (because name clash prevention isn't yet implemented for constructors).
  • Fixed reflection on .NET types to ignore newslot methods that hide a virtual method in a base class.
  • Fixed reflection on .NET types to not emit a public for explicitly implemented interface methods if a base class already has a suitable method.
  • Removed HideFromJavaAttribute from remapped types.
  • Added HideFromJavaAttribute to nested helper type generated for remapped interfaces.

New snapshots: just the binaries and source plus binaries.

Thursday, 04 November 2004 14:13:12 (W. Europe Standard Time, UTC+01:00)  #    Comments [3]
# Tuesday, 19 October 2004
Stack Traces & Line Numbers

Today's snapshot contains a very helpful new feature: Line numbers in stack traces. Unfortunately you only get line numbers in statically compiled code on .NET 1.1, but on Whidbey you'll also get line numbers for dynamically compiled code. At the moment Mono doesn't support them at all :-(

Together with this I also fixed up the stack trace cleaning code to remove most of the IKVM and .NET implementation artifacts from the stack traces. If you don't want the sanitized stack traces, you can use -Dikvm.cleanstacktrace=0 to disable it. This can be helpful to understand what's going on under the covers, especially if there is a bug in the cleanup code ;-)

Here is an example of a cleaned stack trace with line numbers (when run on Whidbey):

public class test
{
  public static void main(String[] args)
  {
    try {
      Object o = new Object();
      ((cli.System.Object)o).ToString();
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      new java.util.Hashtable().put(null, null);
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      "".endsWith(null);
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      "".charAt(1);
    } catch(Throwable t) {
      t.printStackTrace();
    }
  }
}
C:\tmp>ikvm test
java.lang.ClassCastException
        at test.main (test.java:7)
java.lang.NullPointerException
        at java.util.Hashtable.hash (Hashtable.java:822)
        at java.util.Hashtable.put (Hashtable.java:432)
        at test.main (test.java:12)
java.lang.NullPointerException
        at cli.System.String.EndsWith (Unknown Source)
        at java.lang.String.endsWith (map.xml:335)
        at test.main (test.java:17)
java.lang.StringIndexOutOfBoundsException
        at java.lang.String.charAt (map.xml:266)
        at test.main (test.java:22)

What's new:

  • Merged with Classpath cvs.
  • Added stub to AWT toolkit for getClasspathTextLayoutPeer.
  • Added two new public APIs to the runtime:
    - IKVM.Runtime.Util.GetClassFromObject(). It returns the java.lang.Class object corresponding to the passed in object. It is now used by java.lang.Object.getClass().
    - IKVM.Runtime.Util.GetClassFromTypeHandle(). This is used by the compiler when it needs the class object (for synchronized static methods, for static JNI methods and for the new 1.5 ldc class literal).
  • Changed map.xml String.hashCode body to redirect.
  • Inlined String.charAt in map.xml.
  • Improved stack trace cleaning so that IKVM implementation artifacts are removed from exception stack traces.
  • Code compiled with ikvmc now produces stack traces with line numbers even if the -debug option isn't used. On Whidbey, dynamically converted code will also have line numbers in the stack traces.
  • Moved my system/extension class loader implementation into GNU Classpath.
  • Implemented stack walking for AccessController security checks.
  • Fixed -D option. I used a StringDictionary to store the key/value pairs, but that converts the keys to lowercase.
  • Added -nostacktraceinfo option to ikvmc, to disable generating the line number information table and source file attributes.
  • Made ikvmc a little more robust in dealing with bogus path arguments.
  • Changed TypeWrapper.IsAbstract and ClassFile.IsAbstract to return true for interfaces, even if the abstract bit isn't set.
  • Made class loading error tracer more robust.
  • Introduced a CountingILGenerator wrapper for ILGenerator that keeps track of the IL offset (required for the line number tables).
  • Added AttributeHelper methods to set the new SourceFileAttribute and LineNumberTableAttribute.
  • Removed TypeWrapper.PackageName (was only used in one place and it wasn't correct for array types).
  • De-virtualized TypeWrapper.EmitBox (custom boxing support was previously removed).
  • Reflection now reports a public default constructor on all ValueTypes (on the .NET the default constructor for value types is implicit).
  • Finally completed the inheritance inversion for java.lang.Object/cli.System.Object and java.lang.Throwable/cli.System.Exception. Casting and instanceof tests on cli.System.Object and cli.System.Exception will now work consistently with reflection (i.e. cli.System.Object is a subclass of java.lang.Object and cli.System.Exception is a subclass of java.lang.Throwable). Similar work still needs to be done for java.lang.Comparable/cli.System.IComparable and (a little different) java.lang.String/cli.System.String.
  • Introduced two new custom attributes SourceFileAttribute and LineNumberTableAttribute.
  • Added line number tracking to map.xml parsing.
  • Added blt, ldc_i4_1, add and mul opcodes to remapper.
  • Fixed bug in remapper that caused local variables and labels outside of exception blocks not to be visible inside exception blocks.
  • Fixed verifier to detect trying to instantiate an array type with the new bytecode.
  • Fixed verifier bug that caused NullReferenceException on methods that had dead code that contain local variable stores when compiling with the -debug option.
  • Fixed bug in remapped type code generator that caused remapping helper methods to show up in Java reflection (I forgot to emit the HideFromJavaAttribute).
  • Added error handling for unknown attributes to map.xml parser.

New snapshots: just the binaries and source plus binaries.

Tuesday, 19 October 2004 16:10:46 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Thursday, 07 October 2004
Generic Algorithms Revisited

OSNews has an article about using generics to do calculations on Whidbey. When I originally wrote about this, my trick was much slower than the virtual dispatch one that Anders suggested, but it's nice to see that the CLR perf team paid attention, because on the current Whidbey beta it's really fast.

Thursday, 07 October 2004 11:10:03 (W. Europe Daylight Time, UTC+02:00)  #    Comments [3]
# Tuesday, 05 October 2004
Compiling javac into an executable with ikvmc

Just a quick tip: If you want to use javac with IKVM that's possible. The only thing to watch out for is that javac requires access to the class files for the core Java classes. IKVM doesn't include these files, but you can generate them by running ikvmstub on IKVM.GNU.Classpath.dll or you can just let javac use the rt.jar of the JDK.

(The following examples assume that the Java SDK is installed in \j2sdk1.4.0)

Here's how you can run javac with ikvm (wrapped for readability):

ikvm -Dsun.boot.class.path=\j2sdk1.4.0\jre\lib\rt.jar
     -classpath \j2sdk1.4.0\lib\tools.jar
     com.sun.tools.javac.Main
     test.java

Here's how you can compile javac into a .NET executable:

ikvmc -out:javac.exe
      -main:com.sun.tools.javac.Main
      -Dsun.boot.class.path=\j2sdk1.4.0\jre\lib\rt.jar
      \j2sdk1.4.0\lib\tools.jar

Note that this hardcodes the path to rt.jar into the javac.exe file, so this executable can only be used on the system it was created on. Note also that you cannot redistribute the javac executable because this would violate the Sun license.

Tuesday, 05 October 2004 16:17:23 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
New Snapshot

A couple of bug fixes and some cleanup, but the two major new features are the support for Whidbey (ikvmstub can now handle Whidbey's mscorlib) and the fact that getClassLoader() now returns the system class loader for statically compiled classes (instead of null).

Note that Whidbey support only means that IKVM will now run successfully on Whidbey, not that any of the new Whidbey features are available to use from Java. In particular, generic types and generic methods are not visible to Java code.

What's new:

  • Implemented support for assertions.
  • Changed TimeZone code to find an existing Java timezone that matches the current .NET TimeZone (a little better than before but it's still a very lame solution).
  • Added -r short form of -reference option to ikvmc.
  • Improved error handling for incorrect filenames/paths in ikvmc.
  • Changed TypeWrapper.GetFieldWrapper() to take a signature string instead of a TypeWrapper.
  • Statically compiled code (outside of IKVM.GNU.Classpath.dll) now returns the system class loader for Class.getClassLoader(), instead of null.
  • Changed FieldWrapper implementation to subclass based model instead of delegating to CodeEmitter instances.
  • Removed most of the CodeEmitter functionality.
  • Fixed field reflection bug (reflecting on fields in statically compiled Java types or .NET types didn't work unless you had previously reflected on the methods of the type, or dynamically compiled code against the type).
  • Added preliminary support to run on Whidbey (generic types and methods will not be visible from Java).
  • Removed map.xml support for without sig attribute.
  • Fixed reflection on .NET types to ignore methods with ByRef pointer arguments.
  • Added support to ikvmstub to skip .NET types that aren't visible from Java.

New snapshots: just the binaries and source plus binaries.

Update: I updated the snapshot to fix a regression when accessing fields where the field type class is not found.

Tuesday, 05 October 2004 13:32:00 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Monday, 27 September 2004
New Snapshot

A couple of fixes and performance enhancements for static synchronized and static native methods.

Changes:

  • Fixed ikvmc so it no longer throws an exception if no output filename is specified (or implied).
  • Fixed monitorenter instruction and synchronized methods to not throw InterruptedException.
  • Added [DebuggerStepThroughAttribute] to all methods in ByteCodeHelper that didn't have it yet.
  • Added class object caching for static synchronized and native methods.
  • Fixed Thread.stop() so that it works on suspended threads.
  • Implemented a few more random awt peer methods.
  • Removed gnu.classpath.RawData mapping from map.xml.
  • Changed ikvm.exe setting of gnu.classpath.home and java.class.path properties so that they end up in the default system properties.
  • Added IKVM.Runtime.Startup.EnterMainThread() and ExitMainThread() methods (and changed ikvm.exe) to encapsulate the setup and tear-down of the main thread.
  • Fixed JNIEnv.MonitorEnter to use ByteCodeHelper.monitorenter.
  • Fixed compiler bug that caused invoking CharSequence methods to generate a ClassCastException.
  • Changed ikvmc main stub method to call Startup.EnterMainThread() and ExitMainThread() and to pass unhandled exceptions to ThreadGroup.uncaughtException().
  • Fixed JNI method invocation to not wrap exceptions in java.lang.reflect.InvocationTargetException.
  • Fixed support for synchronized native methods.
  • Fixed bug in JNI proxy generator (only used when -Xsave option is specified).
  • Named the WaitShutdownHook and SaveAssemblyShutdownHook threads in starter.cs.
  • Fixed JNIEnv.RegisterNatives when JNI proxy generator is used.
  • Fixed Double.parseDouble to parse +Infinity correctly.

New snapshots: just the binaries and source plus binaries.

Monday, 27 September 2004 12:27:17 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 15 September 2004
New Snapshot

A new snapshot with some minor changes. In the previous entry I forgot to document two JNI restrictions:

  • JNI doesn't work with Mono 1.0 on Windows (Mono doesn't do correct P/Invoke name mangling on Windows -- this can be worked around by rebuilding IKVM.Runtime.dll with the alternative DllImport attributes in JniInterface.cs).
  • JNI libraries are never unloaded. Class loaders aren't garbage collected and consequently, native libraries aren't either.

Changes:

  • Implemented support for globbing on Windows. Both to ikvm.exe and to executables compiled with ikvmc (on ikvmc it can be disabled with the -noglobbing option).
  • Added IKVM.Runtime.Startup class that contains globbing support and a method to set Java system properties.
  • Added option to ikvmc to set Java system properties before starting the application (executables only).
  • Disabled freeing of JNIEnv unmanaged memory (for the time being) because it causes crashes on Mono (possibly due to a GC bug in Mono).
  • Added optimization to mark "side effect free" static initializers with the beforefieldinit flag. (Static compilation only.)
  • Added special case support to the verifier for the "side effect free" optimization.
  • Fixed a verifier bug that caused unverifiable code to be generated in rare circumstances (the code worked OK, it just wasn't verifiable). This was the issue that caused Xerces' org.apache.xerces.util.DOMUtil.copyInto() to fail verification.

New snapshots: just the binaries and source plus binaries.

Wednesday, 15 September 2004 15:42:50 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Thursday, 09 September 2004
JNI Code Complete & A New Snapshot

JNI is now done. Almost complete support for all J2SE 1.4 JNI functionality. There are a few (permanent) restrictions:

  • It's much slower than on a real JVM (most operations are between 10x and 100x slower).
  • You cannot create a string instance with AllocObject and then call the constructor on it with CallNonvirtualVoidMethod. This is unlikely to be a problem, because there really is no reason to go through all this trouble to create a string. The normal ways to create a strings are NewString, NewStringUTF and NewObject and these work fine. If any app were found that depended on two step string construction, a workaround for that particular app could probably be added.
  • JVM.DLL, which implements the three static invocation API functions, only works on Win32.
  • JNI_CreateJavaVM only supports the JDK 1.2 style init args and only the -D option (to define Java properties).
  • JNI_GetDefaultJavaVMInitArgs is not implemented. It's only used for JDK 1.1 style JVM creation, so I don't think it'll be used anymore. If anyone requires it, it could be implemented.
  • JNI_GetCreatedJavaVMs only returns the JavaVM if the VM was started through JNI or a JNI call that retrieves the JavaVM has already occurred.
  • JavaVM.DestroyJVM is only partially implemented (it waits until there are no more non-daemon Java threads and then returns JNI_ERR).
  • JavaVM.DetachCurrentThread doesn't release monitors held by the thread.

JVM.DLL is really cool. It's a small stub created with ILASM. It's only a few lines of code and all it really does is export three managed functions to the unmanaged world. The ability to export managed functions is is a very nice CLR feature that is relatively unknown. On a Win32 system, any native application that hosts the Sun Java Virtual Machine can now use IKVM (without modifying the native app), simply by copying the IKVM DLLs into the native applications directory and removing the real Java VM JVM.DLL from the PATH.

Changes:

  • Imported new Classpath TimeZone handling. This still leaves TimeZone support in a very broken state, but at least the Classpath VM interface is now more powerful and should hopefully allow a correct implementation when I get around to it.
  • Finally added namespaces to all public classes in IKVM.Runtime.dll. There are IKVM.Attributes, IKVM.Runtime, IKVM.Internal and IKVM.NativeCode. IKVM.Internal contains public types that really shouldn't be public, but have to be public because ikvm or ikvmc depend on them. In IKVM.NativeCode live the implementations of the GNU Classpath native methods.
    Don't use the types in the IKVM.Internal or IKVM.NativeCode namespace in your own projects, because they are likely to go away or change in incompatible ways. IKVM.Attributes and IKVM.Runtime are going to be relatively stable (but they can still change up until the 1.0 release) .
  • Added support for JNI thread detaching and waiting for all non-daemon threads to end to VMThread.java.
  • Added GetClassFromTypeHandle method to ByteCodeHelper, this replaces the direct use of NativeCode.java.lang.VMClass.getClassFromType by JNI method stubs and synchronized static methods.
  • Moved the various arraycopy methods from NativeCode.java.lang.VMSystem to ByteCodeHelper.
  • Added getSystemClassLoader method to ClassLoaderWrapper.
  • JavaException.cs: Removed message from ArrayStoreException and added InstantiationException.
  • Added invocation API support to JNI.
  • Renamed JniFrame to Frame and made it an innerclass of IKVM.Runtime.JNI.
  • Implemented more efficient way to determine active JNI method class loader in JNIEnv.FindClass.
  • Fixed JNI method resolution to look for short and long method names in each native library (previous it would first search all libraries for the short name and then in another pass would look for the long names).
  • Added support for JavaVMAttachArgs to JavaVM.AttachCurrentThread.
  • Implemented JavaVM.DestroyJavaVM.
  • Implemented JavaVM.DetachCurrentThread.
  • Implemented JavaVM.AttachCurrentThreadAsDaemon.
  • Implemented JNIEnv cleanup when thread dies.
  • Fixed JNIEnv.DefineClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.FindClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.SetPendingException to do exception mapping and set the stack trace.
  • Fixed JNIEnv.ThrowNew to decode message as platform specific instead of UTF-8.
  • Fixed JNIEnv.ExceptionDescribe to clear pending exception (and to actually work).
  • Fixed JNIEnv.FatalError to decode message as platform specific instead of UTF-8.
  • Added error handling to JNIEnv.AllocObject (if you try to create interface or abstract class instance).
  • Added error handling and value type support to JNIEnv.NewObjectArray.
  • Added value type support to JNIEnv.GetObjectArrayElement and JNIEnv.SetObjectArrayElement.
  • Added error handling to the various JNIEnv.New<PrimitiveType>Array functions.
  • Added error handling to the various JNIEnv.Get<PrimitiveType>ArrayRegion and JNIEnv.Set<PrimitiveType>ArrayRegion functions.
  • Implemented growing the local ref table for native methods that use (or leak) excessive amounts of local refs.
  • Changed stack and local variable type encoding of value types and enums to System.ValueType and System.Enum respectively (instead of simply System.Object).
  • Fixed bug in bytecode compiler that caused it emit incorrect code when calling inherited methods on value types or enums.

New snapshots: just the binaries and source plus binaries.

Thursday, 09 September 2004 13:51:09 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Sunday, 05 September 2004
New Snapshot

A couple of fixes and a clean up of the JNI code. The Managed C++ JNI provider used some clever tricks and when I ported them to C# they turned into ugly hacks, so I removed them. More on that in a future entry.

Changes:

  • Changed VMAccessController and VMClassLoader to use .NET thread local storage instead of a Java ThreadLocal to reduce initialization order dependencies.
  • Added -XXsave flag to ikvm.exe to save debug image of single threaded apps that don't call System.exit().
  • Added -apartment:[sta|mta|none] flag to ikvmc (default is to apply the STAThreadAttribute to main).
  • Added Win64 support to ikvm-native.
  • Fixed class file parser to properly resize the instruction array. Previously, the compiler got confused when emitting debugging information.
  • Added code to save the assembly used for non-virtual reflective method invocations when -Xsave is used.
  • Introduced JNI type name aliases in JniInterface to make the code more readable.
  • Implemented JNI functions: EnsureLocalCapacity, PushLocalFrame, PopLocalFrame, NewWeakGlobalRef and DeleteWeakGlobalRef.
  • Fixed Debug.Assert in GhostMethodWrapper (MemberWrapper.cs).
  • Fixed JniProxyBuilder to emit ldtoken and pass it as an argument to the real JNI method.
  • Improved error handling in gnu.java.io.EncodingManager.
  • Fixed support for setting the java.library.path property on the ikvm.exe command line.
  • Fixed default java.library.path on Windows.
  • Added workaround for Mono 1.0 bug in Marshal.AllocHGlobal.
  • Fixed delegate reflection bug that caused ikvmstub not to emit the nested Method interface for delegates.
  • Added resource name mangling to make ILDASM/ILASM roundtripping work.

New snapshots: just the binaries and source plus binaries.

Sunday, 05 September 2004 11:54:45 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]