# Thursday, 09 September 2004
JNI Code Complete & A New Snapshot

JNI is now done. Almost complete support for all J2SE 1.4 JNI functionality. There are a few (permanent) restrictions:

  • It's much slower than on a real JVM (most operations are between 10x and 100x slower).
  • You cannot create a string instance with AllocObject and then call the constructor on it with CallNonvirtualVoidMethod. This is unlikely to be a problem, because there really is no reason to go through all this trouble to create a string. The normal ways to create a strings are NewString, NewStringUTF and NewObject and these work fine. If any app were found that depended on two step string construction, a workaround for that particular app could probably be added.
  • JVM.DLL, which implements the three static invocation API functions, only works on Win32.
  • JNI_CreateJavaVM only supports the JDK 1.2 style init args and only the -D option (to define Java properties).
  • JNI_GetDefaultJavaVMInitArgs is not implemented. It's only used for JDK 1.1 style JVM creation, so I don't think it'll be used anymore. If anyone requires it, it could be implemented.
  • JNI_GetCreatedJavaVMs only returns the JavaVM if the VM was started through JNI or a JNI call that retrieves the JavaVM has already occurred.
  • JavaVM.DestroyJVM is only partially implemented (it waits until there are no more non-daemon Java threads and then returns JNI_ERR).
  • JavaVM.DetachCurrentThread doesn't release monitors held by the thread.

JVM.DLL is really cool. It's a small stub created with ILASM. It's only a few lines of code and all it really does is export three managed functions to the unmanaged world. The ability to export managed functions is is a very nice CLR feature that is relatively unknown. On a Win32 system, any native application that hosts the Sun Java Virtual Machine can now use IKVM (without modifying the native app), simply by copying the IKVM DLLs into the native applications directory and removing the real Java VM JVM.DLL from the PATH.

Changes:

  • Imported new Classpath TimeZone handling. This still leaves TimeZone support in a very broken state, but at least the Classpath VM interface is now more powerful and should hopefully allow a correct implementation when I get around to it.
  • Finally added namespaces to all public classes in IKVM.Runtime.dll. There are IKVM.Attributes, IKVM.Runtime, IKVM.Internal and IKVM.NativeCode. IKVM.Internal contains public types that really shouldn't be public, but have to be public because ikvm or ikvmc depend on them. In IKVM.NativeCode live the implementations of the GNU Classpath native methods.
    Don't use the types in the IKVM.Internal or IKVM.NativeCode namespace in your own projects, because they are likely to go away or change in incompatible ways. IKVM.Attributes and IKVM.Runtime are going to be relatively stable (but they can still change up until the 1.0 release) .
  • Added support for JNI thread detaching and waiting for all non-daemon threads to end to VMThread.java.
  • Added GetClassFromTypeHandle method to ByteCodeHelper, this replaces the direct use of NativeCode.java.lang.VMClass.getClassFromType by JNI method stubs and synchronized static methods.
  • Moved the various arraycopy methods from NativeCode.java.lang.VMSystem to ByteCodeHelper.
  • Added getSystemClassLoader method to ClassLoaderWrapper.
  • JavaException.cs: Removed message from ArrayStoreException and added InstantiationException.
  • Added invocation API support to JNI.
  • Renamed JniFrame to Frame and made it an innerclass of IKVM.Runtime.JNI.
  • Implemented more efficient way to determine active JNI method class loader in JNIEnv.FindClass.
  • Fixed JNI method resolution to look for short and long method names in each native library (previous it would first search all libraries for the short name and then in another pass would look for the long names).
  • Added support for JavaVMAttachArgs to JavaVM.AttachCurrentThread.
  • Implemented JavaVM.DestroyJavaVM.
  • Implemented JavaVM.DetachCurrentThread.
  • Implemented JavaVM.AttachCurrentThreadAsDaemon.
  • Implemented JNIEnv cleanup when thread dies.
  • Fixed JNIEnv.DefineClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.FindClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.SetPendingException to do exception mapping and set the stack trace.
  • Fixed JNIEnv.ThrowNew to decode message as platform specific instead of UTF-8.
  • Fixed JNIEnv.ExceptionDescribe to clear pending exception (and to actually work).
  • Fixed JNIEnv.FatalError to decode message as platform specific instead of UTF-8.
  • Added error handling to JNIEnv.AllocObject (if you try to create interface or abstract class instance).
  • Added error handling and value type support to JNIEnv.NewObjectArray.
  • Added value type support to JNIEnv.GetObjectArrayElement and JNIEnv.SetObjectArrayElement.
  • Added error handling to the various JNIEnv.New<PrimitiveType>Array functions.
  • Added error handling to the various JNIEnv.Get<PrimitiveType>ArrayRegion and JNIEnv.Set<PrimitiveType>ArrayRegion functions.
  • Implemented growing the local ref table for native methods that use (or leak) excessive amounts of local refs.
  • Changed stack and local variable type encoding of value types and enums to System.ValueType and System.Enum respectively (instead of simply System.Object).
  • Fixed bug in bytecode compiler that caused it emit incorrect code when calling inherited methods on value types or enums.

New snapshots: just the binaries and source plus binaries.

Thursday, 09 September 2004 13:51:09 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Sunday, 05 September 2004
New Snapshot

A couple of fixes and a clean up of the JNI code. The Managed C++ JNI provider used some clever tricks and when I ported them to C# they turned into ugly hacks, so I removed them. More on that in a future entry.

Changes:

  • Changed VMAccessController and VMClassLoader to use .NET thread local storage instead of a Java ThreadLocal to reduce initialization order dependencies.
  • Added -XXsave flag to ikvm.exe to save debug image of single threaded apps that don't call System.exit().
  • Added -apartment:[sta|mta|none] flag to ikvmc (default is to apply the STAThreadAttribute to main).
  • Added Win64 support to ikvm-native.
  • Fixed class file parser to properly resize the instruction array. Previously, the compiler got confused when emitting debugging information.
  • Added code to save the assembly used for non-virtual reflective method invocations when -Xsave is used.
  • Introduced JNI type name aliases in JniInterface to make the code more readable.
  • Implemented JNI functions: EnsureLocalCapacity, PushLocalFrame, PopLocalFrame, NewWeakGlobalRef and DeleteWeakGlobalRef.
  • Fixed Debug.Assert in GhostMethodWrapper (MemberWrapper.cs).
  • Fixed JniProxyBuilder to emit ldtoken and pass it as an argument to the real JNI method.
  • Improved error handling in gnu.java.io.EncodingManager.
  • Fixed support for setting the java.library.path property on the ikvm.exe command line.
  • Fixed default java.library.path on Windows.
  • Added workaround for Mono 1.0 bug in Marshal.AllocHGlobal.
  • Fixed delegate reflection bug that caused ikvmstub not to emit the nested Method interface for delegates.
  • Added resource name mangling to make ILDASM/ILASM roundtripping work.

New snapshots: just the binaries and source plus binaries.

Sunday, 05 September 2004 11:54:45 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 31 August 2004
New Snapshot

The biggest change in this version is that the JNI implementation is now unified. It's completely rewritten in unsafe C# and a tiny bit of C (thanks to Zoltan Varga for porting the C part to Linux). This allows the same code to be used on the Microsoft CLR and on Mono. Porting the Managed C++ to C# was an interesting exercise. Because of the repetitive nature of many of the JNI functions, I really missed the C macros I used in the Managed C++ implementation, but more importantly I found that unsafe C# tries really hard to get in your way. It tries to prevent you from shooting yourself in the foot, but IMO it doesn't really succeed at that, but it does make it really hard to do clever things if you know what you're doing.

I also found some methods missing in System.Runtime.InteropServices.Marshal. There is no memcpy, no way to efficiently allocate unmanaged memory (both CoTaskAllocMem and AllocHGlobal offer functionality or guarantees that you often don't need) and there is no way to turn a delegate into an unmanaged method pointer. I had to add the following C method to convert the delegate an unmanaged method pointer:

JNIEXPORT void* JNICALL ikvm_MarshalDelegate(void* p)
{
    return p;
}

This looks like a no-op, but the P/Invoke marshaler converts the delegate to a pointer to an unmanaged thunk. Unfortunately this functionality is not exposed as a managed API.

Here's what else is new:

  • New unified JNI implementation written mostly in (unsafe) C#. The same JNI implementation now works on Mono and the Microsoft CLR and there is only a tiny bit of platform specific C code (no more Managed C++).
  • Statically compiled code that uses JNI is now portable between different platforms (obviously only if the native library is available on those platforms).
  • Implemented more JNI functionality.
  • Synchronized with GNU Classpath cvs.
  • Fixed Runtime.totalMemory() to return more than Runtime.freeMemory(). Note that the value is still pretty meaningless.
  • Added more validation to BigEndianBinaryReader to better detect invalid class files.
  • Changed ByteCode, NormalizedByteCode and ByteCodeMode enums to be byte based, for efficiency.
  • Separated wide "addressing" modes into ByteCodeModeWide to make the code simpler.
  • Added ByteCodeFlags enum to encode bytecode properties in ByteCodeMetaData.
  • Fixed ByteCodeMetaData encoding for the [ailfd]load/[ailfd]store instructions to support the wide encodings.
  • Made class file parsing eager, to prevent depending on the passed in byte array after ClassLoader.defineClass() returns.
  • Made class file parsing robust against class loaders that to attact the VM by modifying the class file byte array, while it is being parsed.
  • Made many Optimizations to class file parsing.
  • Added more validation to class file parsing.
  • Fixed class file parsing to throw a ClassFormatError when a class loader is trying to redefine java.lang.Object.
  • Implemented class loader namespace support for native libraries.
  • Fixed FieldWrapper.Link() to rethrow exception if linking fails (instead of silently swallowing it).
  • Made the __ static initialization trigger field protected instead of public.
  • Use a more efficient CIL sequence for instanceof instruction.
  • Fixed stupid bug in Double/Float.parse(). It used the local sensitive version of System.Double.Parse().
  • Fixed java.io.System.setErr().

New snapshots: just the binaries and source plus binaries.

Update: I forgot to thank Zoltan Varga for porting the platform specific part of the new JNI implementation to Linux.

Tuesday, 31 August 2004 13:55:43 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 20 August 2004
Non-virtual Method Invocation

In Tuesday's snapshot I finally implemented the CallNonvirtualMethod JNI routines. I've never seen any JNI code that actually uses them, but I obviously needed to implement them at some point.

For those of you unfamiliar with these routines, let me explain them briefly. Normally when you call a virtual method, the runtime type of the instance the method is called on determines the actual method you end up at (so called virtual method dispatch), but these routines allow you to call a base class method as well. So, if you'd use CallNonvirtualObjectMethod on java.lang.Object.toString() on a java.lang.String instance (let's say “foo“), the result you'd get would be java.lang.String@17e809 instead of the string “foo“.

The CLR Reflection API doesn't have the ability to invoke a virtual method non-virtually. Unlike Java however, the CLR does allow the CIL call instruction to be used to call arbitrary methods non-virtually (the Java invokespecial method can only be used to call methods in the same class or in a base class). So the obvious way to implement these routines, would seem to be to dynamically emit some code that calls the requested method non-virtually, but there is problem with this approach, the JNI routines don't enforce accessibility (they allow you to call non-public methods) and the CLR enforces accessibility. I got around this by using RuntimeMethodHandle.GetFunctionPointer() incombination with the CIL calli instruction. This also has the advantage that the dynamically emitted stubs can be reused for methods with the same signature, because the target address isn't baked into the method, but passed in.

Non-orthogonality

While I don't have a very compelling use case for non-virtual method invocation through reflection, based on API orthogonality and completeness a case could be made for it, but more importantly, I think that ECMA/Microsoft should consider changing the CLI spec to allow non-verifiable code to bypass the accessibility checks. Note that the new DynamicMethod in Whidbey already allows accessibility checks to be bypassed (and note also that whoever wrote the doc for DynamicMethod doesn't understand the difference between visibility and accessibility).

The trick of using calli to bypass the accessibility checks doesn't work for virtual method invocation, because there is no way to dynamically and efficiently get the function pointer of a method by applying the virtual method dispatch rules (i.e. there is no dynamic equivalent of the CIL ldvirtftn instruction).

To summarize, here is what I think should be changed:

  • CLR shouldn't bother enforcing accessibility in non-verifiable code.
  • MethodInfo should get a new InvokeNonvirtual method.
  • MethodInfo should get a GetFunctionPointer(object) method that returns the method pointer that results from virtual dispatch.
Friday, 20 August 2004 09:11:25 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Thursday, 19 August 2004
Article on IKVM

Avik Sengupta wrote an excellent article on IKVM.

Thursday, 19 August 2004 13:33:47 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 17 August 2004
New Snapshot

A new development snapshot. I made many changes (way too many) since the last snapshot. The diff was over 13,000 lines (and I had already checked in part of the changes).

This is an important step towards 1.0, but there is still much to be done.

BTW, Eclipse 3.0 starts up with this version, but that's pretty much it, it doesn't do anything useful yet.

What's New:

  • Changed version numbers to 0.9.* (I'm using a Linux kernel version scheme, even numbers are used for releases, odd numbers are used during development.)
  • Resynchronized with GNU Classpath cvs.
  • Made String.compareTo(Object) redirect to my own implementation instead of System.String.CompareTo (including when the method is invoked through the java.lang.Comparable interface, which is still remapped to System.IComparable).
  • Fixed exception remapping handling of java.lang.ExceptionInInitializer.
  • Added dummy (but safe) implementations of getHostAddress and hostsEqual to ikvmres protocol handler.
  • Removed work around for Classpath serialization bug from Throwable cause serialization. The state of whether the cause was set is now properly serialized.
  • Fixed deserialization of exceptions serialized with pre-1.4 JDKs.
  • Fixed integer overflow bugs in the various argument tests in the String(byte[], ...) constructors.
  • Added missing argument check to String.startsWith(String prefix, int offset).
  • Added -Xbreak option to ikvm, to fire the debugger on startup.
  • Added support for automatically generating build and revision numbers to ikvmc (e.g. -version:0.9.*)
  • Fixed ikvmstub bug that caused invalid max_locals to be set in constructor definitions if the constructor took long or double arguments.
  • Fixed visibility handling of nested types. A public nested type nested inside a non-public type was incorrectly treated as public.
  • Fixed reflection bug. Statically compiled classes had private methods incorrectly reported as final.
  • Fixed method override resolver bug. Having a private final method in a base class with a similar name/signature as a virtual method in the subclass caused a VerifyError (which was presented as a critical failure).
  • Fixed another method override resolver bug. Only interfaces implemented by direct base class were searched, not those implemented by all base classes.
  • Added workaround for deserializing incorrectly serialized java.lang.String objects (strings are supposed to be special cased by serialization, but if you manually construct a serialization stream that contains a TC_OBJECT of class java.lang.String, previously you'd get a weird exception.)
  • Added (untested) implementation of VMAccessController.
  • Fixed small compiler bug. Ldarg/Starg instruction was emitted with int argument, instead of short. This caused two bogus nops in the CIL stream to follow each Ldarg/Starg instruction (in exceptional cases).
  • Fixed Tracer to handle messages without formatting characters.
  • Fixed lame typo in JVM.CompilerClassLoader.EmitRedirect. (Call to Debug.CompareTo(...) instead of System.Diagnostics.Debug.Assert(...)).
  • Added nonvirtualAlternateBody support to map.xml parser.
  • Added nonvirtualAlternateBody to map.xml for java.lang.Object.toString().
  • Added exceptionBlock support to map.xml parser.
  • Added exception handler to map.xml for java.lang.ThreadDeath exception mapping code, to catch ThreadStateException if Thread.Abort is called while it isn't required.
  • Removed unnecessary path name check from FileChannelImpl.open().
  • Removed VMFile.demangle(). Normalization is now part of Classpath java.io.File.
  • Added exception handling to VMFile.isHidden().
  • Added VMFile.toCanonicalForm(). Now used by Classpath java.io.File.getCanonicalPath().
  • Added protection domain support to VMClass.createClass().
  • Removed VMClass.loadBootstrapClass().
  • Removed unused deprecated version of VMClassLoader.defineClass().
  • Added bootstrap classloader resource support to VMClassLoader.
  • Implemented VMClassLoader.getSystemClassLoader() in VMClassLoader. The system classloader is now always available, not just if the application was started with ikvm.exe.
  • Added app.config appSettings support to VMRuntime.insertSystemProperties(). Java properties can now be specified in your .NET application .config file:
    <appSettings>
       <add key="ikvm:<propertyname>" value="<propertyvalue>" />
    </appSettings>
  • In VMThread, removed cleanupDataStoreSlot static and always do lookup of the LocalDataStoreSlot by name, to handle scenario where the IKVM main thread is the first one to lookup the LocalDataStoreSlot (and previously that would cause the static initializer of VMThread to fail, because AllocateNamedDataSlot would fail, as the name was already in use).
  • Fixed Field.checkAccess() when caller is a global method.
  • Removed ExtClassLoader and AppClassLoader from starter.cs (ikvm.exe).
  • Added hack to set the priority of the ikvmdump.exe writing thread to AboveNormal, to deal with a spinning thread in Eclipse on shutdown.
  • Don't add CLASSPATH to java.class.path is the -jar option is specified.
  • Removed support for DLLs in the bootclasspath.
  • Fixed exception handling compilation options for IKVM.JNI.CLR-Win32 in clr-win32.build.
  • Major rewrite of local and global ref support in Win32 JNI provider.
  • Implemented most of the remaining JNI functions in Win32 JNI provider.
  • Fixed class file parser to disallow native constructors.
  • Rewrote class loading and method/field linking in ClassFile.cs.
  • Fixed most of the thread safety issues in class loading/reflection.
  • The runtime no longer throws Java exceptions, instead it throws its own exception, which are converted to Java exceptions at the runtime/Java boundaries. This makes ikvmc compilation of Classpath more robust (previously, if an exception required by the runtime was missing, it would often result in infinite recursion).
  • Simplified MethodDescriptor. It is now only a name/signature tuple.
  • MethodWrapper now uses virtual methods instead of delegates to emit call/callvirt/newobj sequences.
  • Major rewrite of method resolving/linking and MethodWrapper infrastructure. Loader constraints are now properly enforced and LinkageErrors are thrown in the right places at the right times.
  • Renamed HideFromReflectionAttribute to HideFromJavaAttribute and made the semantics consistent. It now means that the method is totally invisible from Java (either through reflection or statically). The new MirandaMethodAttribute signals that a method is invisible from reflection, but callable by Java byte code.
  • Added new NameSigAttribute that is used to recover the original method name or signature when a method name is mangled or when a signature is changed in such a way that at runtime the original signature cannot be reconstructed anymore.
  • Fixed TypeWrapper.IsInSamePackage() to understand that a package with the same name but in different assemblies, really isn't the same package.
  • Fixed ininst instruction in map.xml. Now works the same for Java and .NET types.

I'm sure I forgot many changes/fixes. This change set was just too large. I promise to check in more frequently in future :-)

New snapshots: just the binaries and source plus binaries.

Tuesday, 17 August 2004 11:08:03 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Monday, 16 August 2004
JavaScript on IKVM

Phil blogs about running Rhino on IKVM on Mono.

Monday, 16 August 2004 17:26:15 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 10 August 2004
Whidbey, Generics, Co- and contravariance

I came across an interesting Whidbey beta 1 CLR feature today. As far as I can tell this hasn't been documented anywhere yet, but the Whidbey beta 1 CLR supports covariant and contravariant generic type parameters.

This is a really interesting feature. If C# would support it (at the moment it doesn't), you'd be able to do something like this:

interface IReader<T>
{
  T Get(); 
}
interface IWriter<T>
{
  void Put(T t);
}
class Test<T> : IReader<T>, IWriter<T>
{
  T t;
  public T Get() { return t; }
  public void Put(T t) { this.t = t; }
}
class Driver
{
  static void Main()
  {
    Test<string> t1 = new Test<string>();
    t1.Put("covariance");
    IReader<object> rdr = t1;
    Console.WriteLine(rdr.Get());
    Test<object> t2 = new Test<object>();
    IWriter<string> wrtr = t1;
    wrtr.Put("contravariance");
    Console.WriteLine(t2.Get());
  }
}

Notice that in the first highlighted line we're assigning an object reference that implements IReader<string> to a local variable of type IReader<object>. This is covariance and it is type safe because the IReader<T> interface only returns T. In the second highlighted line we assign a reference that implements IWriter<object> to a local of type IWriter<string>, this is contravariance and this is also type safe, because IWriter<T> only accept arguments of type T.

Unfortunately, C# doesn't yet seem to support this feature, but if we manually write the IL, it'll run on the Whidbey beta 1 CLR (and it is fully verifiable):

.assembly extern mscorlib {}
.assembly test {}

.class interface public abstract
      'IReader`1'<+([mscorlib]System.Object) T>
{
  .method public abstract instance
          !T Get() cil managed {}
}

.class interface public abstract
      'IWriter`1'<-([mscorlib]System.Object) T>
{
  .method public abstract instance
          void Put(!T t) cil managed {}
}

.class public 'Test`1'<([mscorlib]System.Object) T>
      extends [mscorlib]System.Object
      implements class 'IReader`1'<!T>,
                 class 'IWriter`1'<!T>
{
  .field private !T v

  .method public virtual instance
          !T Get() cil managed
  {
    ldarg.0
    ldfld !0 class 'Test`1'<!T>::v
    ret
  }

  .method public virtual instance
          void Put(!T t) cil managed
  {
    ldarg.0
    ldarg.1
    stfld !0 class 'Test`1'<!T>::v
    ret
  }

  .method public specialname rtspecialname
          instance void .ctor() cil managed
  {
    ldarg.0
    call instance void [mscorlib]System.Object::.ctor()
    ret
  }
}

.method static void Main(string[] args) cil managed
{
  .entrypoint
  .locals init ([0] class 'Test`1'<string> t1,
                [1] class 'IReader`1'<object> rdr,
                [2] class 'Test`1'<object> t2,
                [3] class 'IWriter`1'<string> wrtr)

  newobj instance void class 'Test`1'<string>::.ctor()
  stloc.0
  ldloc.0
  ldstr "covariance"
  callvirt instance void class 'Test`1'<string>::Put(!0)
  ldloc.0
  stloc.1
  ldloc.1
  callvirt instance !0 class 'IReader`1'<object>::Get()
  call void [mscorlib]System.Console::WriteLine(object)

  newobj instance void class 'Test`1'<object>::.ctor()
  stloc.2
  ldloc.2
  stloc.3
  ldloc.3
  ldstr "contravariance"
  callvirt instance void class 'IWriter`1'<string>::Put(!0)
  ldloc.2
  callvirt instance !0 class 'IReader`1'<object>::Get()
  call void [mscorlib]System.Console::WriteLine(object)
  ret
}
 

Notice the plus and minus signs in the interface declarations to signal that a generic type parameter is covariant or contravariant.

Tuesday, 10 August 2004 16:03:46 (W. Europe Daylight Time, UTC+02:00)  #    Comments [6]
# Wednesday, 30 June 2004
Mono 1.0

Congratulations to the Mono team with the release of 1.0!

Wednesday, 30 June 2004 19:03:12 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
Finally a Real Web Site

Many thanks to Brian J. Sletten and Stephen Schaub for building the new IKVM.NET web site. I will continue to write this blog, but from now on instructions, documentation and howto articles will appear on the new site (in addition to the excellent content that is already there), where they'll have a more permanent home than here on the blog.

Update: The samples made by Brian and Stephen are available for download on SourceForge now as well.

Wednesday, 30 June 2004 14:34:50 (W. Europe Daylight Time, UTC+02:00)  #    Comments [3]