# Tuesday, 31 August 2004
New Snapshot

The biggest change in this version is that the JNI implementation is now unified. It's completely rewritten in unsafe C# and a tiny bit of C (thanks to Zoltan Varga for porting the C part to Linux). This allows the same code to be used on the Microsoft CLR and on Mono. Porting the Managed C++ to C# was an interesting exercise. Because of the repetitive nature of many of the JNI functions, I really missed the C macros I used in the Managed C++ implementation, but more importantly I found that unsafe C# tries really hard to get in your way. It tries to prevent you from shooting yourself in the foot, but IMO it doesn't really succeed at that, but it does make it really hard to do clever things if you know what you're doing.

I also found some methods missing in System.Runtime.InteropServices.Marshal. There is no memcpy, no way to efficiently allocate unmanaged memory (both CoTaskAllocMem and AllocHGlobal offer functionality or guarantees that you often don't need) and there is no way to turn a delegate into an unmanaged method pointer. I had to add the following C method to convert the delegate an unmanaged method pointer:

JNIEXPORT void* JNICALL ikvm_MarshalDelegate(void* p)
{
    return p;
}

This looks like a no-op, but the P/Invoke marshaler converts the delegate to a pointer to an unmanaged thunk. Unfortunately this functionality is not exposed as a managed API.

Here's what else is new:

  • New unified JNI implementation written mostly in (unsafe) C#. The same JNI implementation now works on Mono and the Microsoft CLR and there is only a tiny bit of platform specific C code (no more Managed C++).
  • Statically compiled code that uses JNI is now portable between different platforms (obviously only if the native library is available on those platforms).
  • Implemented more JNI functionality.
  • Synchronized with GNU Classpath cvs.
  • Fixed Runtime.totalMemory() to return more than Runtime.freeMemory(). Note that the value is still pretty meaningless.
  • Added more validation to BigEndianBinaryReader to better detect invalid class files.
  • Changed ByteCode, NormalizedByteCode and ByteCodeMode enums to be byte based, for efficiency.
  • Separated wide "addressing" modes into ByteCodeModeWide to make the code simpler.
  • Added ByteCodeFlags enum to encode bytecode properties in ByteCodeMetaData.
  • Fixed ByteCodeMetaData encoding for the [ailfd]load/[ailfd]store instructions to support the wide encodings.
  • Made class file parsing eager, to prevent depending on the passed in byte array after ClassLoader.defineClass() returns.
  • Made class file parsing robust against class loaders that to attact the VM by modifying the class file byte array, while it is being parsed.
  • Made many Optimizations to class file parsing.
  • Added more validation to class file parsing.
  • Fixed class file parsing to throw a ClassFormatError when a class loader is trying to redefine java.lang.Object.
  • Implemented class loader namespace support for native libraries.
  • Fixed FieldWrapper.Link() to rethrow exception if linking fails (instead of silently swallowing it).
  • Made the __ static initialization trigger field protected instead of public.
  • Use a more efficient CIL sequence for instanceof instruction.
  • Fixed stupid bug in Double/Float.parse(). It used the local sensitive version of System.Double.Parse().
  • Fixed java.io.System.setErr().

New snapshots: just the binaries and source plus binaries.

Update: I forgot to thank Zoltan Varga for porting the platform specific part of the new JNI implementation to Linux.

Tuesday, 31 August 2004 13:55:43 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 20 August 2004
Non-virtual Method Invocation

In Tuesday's snapshot I finally implemented the CallNonvirtualMethod JNI routines. I've never seen any JNI code that actually uses them, but I obviously needed to implement them at some point.

For those of you unfamiliar with these routines, let me explain them briefly. Normally when you call a virtual method, the runtime type of the instance the method is called on determines the actual method you end up at (so called virtual method dispatch), but these routines allow you to call a base class method as well. So, if you'd use CallNonvirtualObjectMethod on java.lang.Object.toString() on a java.lang.String instance (let's say “foo“), the result you'd get would be java.lang.String@17e809 instead of the string “foo“.

The CLR Reflection API doesn't have the ability to invoke a virtual method non-virtually. Unlike Java however, the CLR does allow the CIL call instruction to be used to call arbitrary methods non-virtually (the Java invokespecial method can only be used to call methods in the same class or in a base class). So the obvious way to implement these routines, would seem to be to dynamically emit some code that calls the requested method non-virtually, but there is problem with this approach, the JNI routines don't enforce accessibility (they allow you to call non-public methods) and the CLR enforces accessibility. I got around this by using RuntimeMethodHandle.GetFunctionPointer() incombination with the CIL calli instruction. This also has the advantage that the dynamically emitted stubs can be reused for methods with the same signature, because the target address isn't baked into the method, but passed in.

Non-orthogonality

While I don't have a very compelling use case for non-virtual method invocation through reflection, based on API orthogonality and completeness a case could be made for it, but more importantly, I think that ECMA/Microsoft should consider changing the CLI spec to allow non-verifiable code to bypass the accessibility checks. Note that the new DynamicMethod in Whidbey already allows accessibility checks to be bypassed (and note also that whoever wrote the doc for DynamicMethod doesn't understand the difference between visibility and accessibility).

The trick of using calli to bypass the accessibility checks doesn't work for virtual method invocation, because there is no way to dynamically and efficiently get the function pointer of a method by applying the virtual method dispatch rules (i.e. there is no dynamic equivalent of the CIL ldvirtftn instruction).

To summarize, here is what I think should be changed:

  • CLR shouldn't bother enforcing accessibility in non-verifiable code.
  • MethodInfo should get a new InvokeNonvirtual method.
  • MethodInfo should get a GetFunctionPointer(object) method that returns the method pointer that results from virtual dispatch.
Friday, 20 August 2004 09:11:25 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Thursday, 19 August 2004
Article on IKVM

Avik Sengupta wrote an excellent article on IKVM.

Thursday, 19 August 2004 13:33:47 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 17 August 2004
New Snapshot

A new development snapshot. I made many changes (way too many) since the last snapshot. The diff was over 13,000 lines (and I had already checked in part of the changes).

This is an important step towards 1.0, but there is still much to be done.

BTW, Eclipse 3.0 starts up with this version, but that's pretty much it, it doesn't do anything useful yet.

What's New:

  • Changed version numbers to 0.9.* (I'm using a Linux kernel version scheme, even numbers are used for releases, odd numbers are used during development.)
  • Resynchronized with GNU Classpath cvs.
  • Made String.compareTo(Object) redirect to my own implementation instead of System.String.CompareTo (including when the method is invoked through the java.lang.Comparable interface, which is still remapped to System.IComparable).
  • Fixed exception remapping handling of java.lang.ExceptionInInitializer.
  • Added dummy (but safe) implementations of getHostAddress and hostsEqual to ikvmres protocol handler.
  • Removed work around for Classpath serialization bug from Throwable cause serialization. The state of whether the cause was set is now properly serialized.
  • Fixed deserialization of exceptions serialized with pre-1.4 JDKs.
  • Fixed integer overflow bugs in the various argument tests in the String(byte[], ...) constructors.
  • Added missing argument check to String.startsWith(String prefix, int offset).
  • Added -Xbreak option to ikvm, to fire the debugger on startup.
  • Added support for automatically generating build and revision numbers to ikvmc (e.g. -version:0.9.*)
  • Fixed ikvmstub bug that caused invalid max_locals to be set in constructor definitions if the constructor took long or double arguments.
  • Fixed visibility handling of nested types. A public nested type nested inside a non-public type was incorrectly treated as public.
  • Fixed reflection bug. Statically compiled classes had private methods incorrectly reported as final.
  • Fixed method override resolver bug. Having a private final method in a base class with a similar name/signature as a virtual method in the subclass caused a VerifyError (which was presented as a critical failure).
  • Fixed another method override resolver bug. Only interfaces implemented by direct base class were searched, not those implemented by all base classes.
  • Added workaround for deserializing incorrectly serialized java.lang.String objects (strings are supposed to be special cased by serialization, but if you manually construct a serialization stream that contains a TC_OBJECT of class java.lang.String, previously you'd get a weird exception.)
  • Added (untested) implementation of VMAccessController.
  • Fixed small compiler bug. Ldarg/Starg instruction was emitted with int argument, instead of short. This caused two bogus nops in the CIL stream to follow each Ldarg/Starg instruction (in exceptional cases).
  • Fixed Tracer to handle messages without formatting characters.
  • Fixed lame typo in JVM.CompilerClassLoader.EmitRedirect. (Call to Debug.CompareTo(...) instead of System.Diagnostics.Debug.Assert(...)).
  • Added nonvirtualAlternateBody support to map.xml parser.
  • Added nonvirtualAlternateBody to map.xml for java.lang.Object.toString().
  • Added exceptionBlock support to map.xml parser.
  • Added exception handler to map.xml for java.lang.ThreadDeath exception mapping code, to catch ThreadStateException if Thread.Abort is called while it isn't required.
  • Removed unnecessary path name check from FileChannelImpl.open().
  • Removed VMFile.demangle(). Normalization is now part of Classpath java.io.File.
  • Added exception handling to VMFile.isHidden().
  • Added VMFile.toCanonicalForm(). Now used by Classpath java.io.File.getCanonicalPath().
  • Added protection domain support to VMClass.createClass().
  • Removed VMClass.loadBootstrapClass().
  • Removed unused deprecated version of VMClassLoader.defineClass().
  • Added bootstrap classloader resource support to VMClassLoader.
  • Implemented VMClassLoader.getSystemClassLoader() in VMClassLoader. The system classloader is now always available, not just if the application was started with ikvm.exe.
  • Added app.config appSettings support to VMRuntime.insertSystemProperties(). Java properties can now be specified in your .NET application .config file:
    <appSettings>
       <add key="ikvm:<propertyname>" value="<propertyvalue>" />
    </appSettings>
  • In VMThread, removed cleanupDataStoreSlot static and always do lookup of the LocalDataStoreSlot by name, to handle scenario where the IKVM main thread is the first one to lookup the LocalDataStoreSlot (and previously that would cause the static initializer of VMThread to fail, because AllocateNamedDataSlot would fail, as the name was already in use).
  • Fixed Field.checkAccess() when caller is a global method.
  • Removed ExtClassLoader and AppClassLoader from starter.cs (ikvm.exe).
  • Added hack to set the priority of the ikvmdump.exe writing thread to AboveNormal, to deal with a spinning thread in Eclipse on shutdown.
  • Don't add CLASSPATH to java.class.path is the -jar option is specified.
  • Removed support for DLLs in the bootclasspath.
  • Fixed exception handling compilation options for IKVM.JNI.CLR-Win32 in clr-win32.build.
  • Major rewrite of local and global ref support in Win32 JNI provider.
  • Implemented most of the remaining JNI functions in Win32 JNI provider.
  • Fixed class file parser to disallow native constructors.
  • Rewrote class loading and method/field linking in ClassFile.cs.
  • Fixed most of the thread safety issues in class loading/reflection.
  • The runtime no longer throws Java exceptions, instead it throws its own exception, which are converted to Java exceptions at the runtime/Java boundaries. This makes ikvmc compilation of Classpath more robust (previously, if an exception required by the runtime was missing, it would often result in infinite recursion).
  • Simplified MethodDescriptor. It is now only a name/signature tuple.
  • MethodWrapper now uses virtual methods instead of delegates to emit call/callvirt/newobj sequences.
  • Major rewrite of method resolving/linking and MethodWrapper infrastructure. Loader constraints are now properly enforced and LinkageErrors are thrown in the right places at the right times.
  • Renamed HideFromReflectionAttribute to HideFromJavaAttribute and made the semantics consistent. It now means that the method is totally invisible from Java (either through reflection or statically). The new MirandaMethodAttribute signals that a method is invisible from reflection, but callable by Java byte code.
  • Added new NameSigAttribute that is used to recover the original method name or signature when a method name is mangled or when a signature is changed in such a way that at runtime the original signature cannot be reconstructed anymore.
  • Fixed TypeWrapper.IsInSamePackage() to understand that a package with the same name but in different assemblies, really isn't the same package.
  • Fixed ininst instruction in map.xml. Now works the same for Java and .NET types.

I'm sure I forgot many changes/fixes. This change set was just too large. I promise to check in more frequently in future :-)

New snapshots: just the binaries and source plus binaries.

Tuesday, 17 August 2004 11:08:03 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Monday, 16 August 2004
JavaScript on IKVM

Phil blogs about running Rhino on IKVM on Mono.

Monday, 16 August 2004 17:26:15 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 10 August 2004
Whidbey, Generics, Co- and contravariance

I came across an interesting Whidbey beta 1 CLR feature today. As far as I can tell this hasn't been documented anywhere yet, but the Whidbey beta 1 CLR supports covariant and contravariant generic type parameters.

This is a really interesting feature. If C# would support it (at the moment it doesn't), you'd be able to do something like this:

interface IReader<T>
{
  T Get(); 
}
interface IWriter<T>
{
  void Put(T t);
}
class Test<T> : IReader<T>, IWriter<T>
{
  T t;
  public T Get() { return t; }
  public void Put(T t) { this.t = t; }
}
class Driver
{
  static void Main()
  {
    Test<string> t1 = new Test<string>();
    t1.Put("covariance");
    IReader<object> rdr = t1;
    Console.WriteLine(rdr.Get());
    Test<object> t2 = new Test<object>();
    IWriter<string> wrtr = t1;
    wrtr.Put("contravariance");
    Console.WriteLine(t2.Get());
  }
}

Notice that in the first highlighted line we're assigning an object reference that implements IReader<string> to a local variable of type IReader<object>. This is covariance and it is type safe because the IReader<T> interface only returns T. In the second highlighted line we assign a reference that implements IWriter<object> to a local of type IWriter<string>, this is contravariance and this is also type safe, because IWriter<T> only accept arguments of type T.

Unfortunately, C# doesn't yet seem to support this feature, but if we manually write the IL, it'll run on the Whidbey beta 1 CLR (and it is fully verifiable):

.assembly extern mscorlib {}
.assembly test {}

.class interface public abstract
      'IReader`1'<+([mscorlib]System.Object) T>
{
  .method public abstract instance
          !T Get() cil managed {}
}

.class interface public abstract
      'IWriter`1'<-([mscorlib]System.Object) T>
{
  .method public abstract instance
          void Put(!T t) cil managed {}
}

.class public 'Test`1'<([mscorlib]System.Object) T>
      extends [mscorlib]System.Object
      implements class 'IReader`1'<!T>,
                 class 'IWriter`1'<!T>
{
  .field private !T v

  .method public virtual instance
          !T Get() cil managed
  {
    ldarg.0
    ldfld !0 class 'Test`1'<!T>::v
    ret
  }

  .method public virtual instance
          void Put(!T t) cil managed
  {
    ldarg.0
    ldarg.1
    stfld !0 class 'Test`1'<!T>::v
    ret
  }

  .method public specialname rtspecialname
          instance void .ctor() cil managed
  {
    ldarg.0
    call instance void [mscorlib]System.Object::.ctor()
    ret
  }
}

.method static void Main(string[] args) cil managed
{
  .entrypoint
  .locals init ([0] class 'Test`1'<string> t1,
                [1] class 'IReader`1'<object> rdr,
                [2] class 'Test`1'<object> t2,
                [3] class 'IWriter`1'<string> wrtr)

  newobj instance void class 'Test`1'<string>::.ctor()
  stloc.0
  ldloc.0
  ldstr "covariance"
  callvirt instance void class 'Test`1'<string>::Put(!0)
  ldloc.0
  stloc.1
  ldloc.1
  callvirt instance !0 class 'IReader`1'<object>::Get()
  call void [mscorlib]System.Console::WriteLine(object)

  newobj instance void class 'Test`1'<object>::.ctor()
  stloc.2
  ldloc.2
  stloc.3
  ldloc.3
  ldstr "contravariance"
  callvirt instance void class 'IWriter`1'<string>::Put(!0)
  ldloc.2
  callvirt instance !0 class 'IReader`1'<object>::Get()
  call void [mscorlib]System.Console::WriteLine(object)
  ret
}
 

Notice the plus and minus signs in the interface declarations to signal that a generic type parameter is covariant or contravariant.

Tuesday, 10 August 2004 16:03:46 (W. Europe Daylight Time, UTC+02:00)  #    Comments [6]