# Tuesday, 19 October 2004
Stack Traces & Line Numbers

Today's snapshot contains a very helpful new feature: Line numbers in stack traces. Unfortunately you only get line numbers in statically compiled code on .NET 1.1, but on Whidbey you'll also get line numbers for dynamically compiled code. At the moment Mono doesn't support them at all :-(

Together with this I also fixed up the stack trace cleaning code to remove most of the IKVM and .NET implementation artifacts from the stack traces. If you don't want the sanitized stack traces, you can use -Dikvm.cleanstacktrace=0 to disable it. This can be helpful to understand what's going on under the covers, especially if there is a bug in the cleanup code ;-)

Here is an example of a cleaned stack trace with line numbers (when run on Whidbey):

public class test
{
  public static void main(String[] args)
  {
    try {
      Object o = new Object();
      ((cli.System.Object)o).ToString();
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      new java.util.Hashtable().put(null, null);
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      "".endsWith(null);
    } catch(Throwable t) {
      t.printStackTrace();
    }
    try {
      "".charAt(1);
    } catch(Throwable t) {
      t.printStackTrace();
    }
  }
}
C:\tmp>ikvm test
java.lang.ClassCastException
        at test.main (test.java:7)
java.lang.NullPointerException
        at java.util.Hashtable.hash (Hashtable.java:822)
        at java.util.Hashtable.put (Hashtable.java:432)
        at test.main (test.java:12)
java.lang.NullPointerException
        at cli.System.String.EndsWith (Unknown Source)
        at java.lang.String.endsWith (map.xml:335)
        at test.main (test.java:17)
java.lang.StringIndexOutOfBoundsException
        at java.lang.String.charAt (map.xml:266)
        at test.main (test.java:22)

What's new:

  • Merged with Classpath cvs.
  • Added stub to AWT toolkit for getClasspathTextLayoutPeer.
  • Added two new public APIs to the runtime:
    - IKVM.Runtime.Util.GetClassFromObject(). It returns the java.lang.Class object corresponding to the passed in object. It is now used by java.lang.Object.getClass().
    - IKVM.Runtime.Util.GetClassFromTypeHandle(). This is used by the compiler when it needs the class object (for synchronized static methods, for static JNI methods and for the new 1.5 ldc class literal).
  • Changed map.xml String.hashCode body to redirect.
  • Inlined String.charAt in map.xml.
  • Improved stack trace cleaning so that IKVM implementation artifacts are removed from exception stack traces.
  • Code compiled with ikvmc now produces stack traces with line numbers even if the -debug option isn't used. On Whidbey, dynamically converted code will also have line numbers in the stack traces.
  • Moved my system/extension class loader implementation into GNU Classpath.
  • Implemented stack walking for AccessController security checks.
  • Fixed -D option. I used a StringDictionary to store the key/value pairs, but that converts the keys to lowercase.
  • Added -nostacktraceinfo option to ikvmc, to disable generating the line number information table and source file attributes.
  • Made ikvmc a little more robust in dealing with bogus path arguments.
  • Changed TypeWrapper.IsAbstract and ClassFile.IsAbstract to return true for interfaces, even if the abstract bit isn't set.
  • Made class loading error tracer more robust.
  • Introduced a CountingILGenerator wrapper for ILGenerator that keeps track of the IL offset (required for the line number tables).
  • Added AttributeHelper methods to set the new SourceFileAttribute and LineNumberTableAttribute.
  • Removed TypeWrapper.PackageName (was only used in one place and it wasn't correct for array types).
  • De-virtualized TypeWrapper.EmitBox (custom boxing support was previously removed).
  • Reflection now reports a public default constructor on all ValueTypes (on the .NET the default constructor for value types is implicit).
  • Finally completed the inheritance inversion for java.lang.Object/cli.System.Object and java.lang.Throwable/cli.System.Exception. Casting and instanceof tests on cli.System.Object and cli.System.Exception will now work consistently with reflection (i.e. cli.System.Object is a subclass of java.lang.Object and cli.System.Exception is a subclass of java.lang.Throwable). Similar work still needs to be done for java.lang.Comparable/cli.System.IComparable and (a little different) java.lang.String/cli.System.String.
  • Introduced two new custom attributes SourceFileAttribute and LineNumberTableAttribute.
  • Added line number tracking to map.xml parsing.
  • Added blt, ldc_i4_1, add and mul opcodes to remapper.
  • Fixed bug in remapper that caused local variables and labels outside of exception blocks not to be visible inside exception blocks.
  • Fixed verifier to detect trying to instantiate an array type with the new bytecode.
  • Fixed verifier bug that caused NullReferenceException on methods that had dead code that contain local variable stores when compiling with the -debug option.
  • Fixed bug in remapped type code generator that caused remapping helper methods to show up in Java reflection (I forgot to emit the HideFromJavaAttribute).
  • Added error handling for unknown attributes to map.xml parser.

New snapshots: just the binaries and source plus binaries.

Tuesday, 19 October 2004 16:10:46 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Thursday, 07 October 2004
Generic Algorithms Revisited

OSNews has an article about using generics to do calculations on Whidbey. When I originally wrote about this, my trick was much slower than the virtual dispatch one that Anders suggested, but it's nice to see that the CLR perf team paid attention, because on the current Whidbey beta it's really fast.

Thursday, 07 October 2004 11:10:03 (W. Europe Daylight Time, UTC+02:00)  #    Comments [3]
# Tuesday, 05 October 2004
Compiling javac into an executable with ikvmc

Just a quick tip: If you want to use javac with IKVM that's possible. The only thing to watch out for is that javac requires access to the class files for the core Java classes. IKVM doesn't include these files, but you can generate them by running ikvmstub on IKVM.GNU.Classpath.dll or you can just let javac use the rt.jar of the JDK.

(The following examples assume that the Java SDK is installed in \j2sdk1.4.0)

Here's how you can run javac with ikvm (wrapped for readability):

ikvm -Dsun.boot.class.path=\j2sdk1.4.0\jre\lib\rt.jar
     -classpath \j2sdk1.4.0\lib\tools.jar
     com.sun.tools.javac.Main
     test.java

Here's how you can compile javac into a .NET executable:

ikvmc -out:javac.exe
      -main:com.sun.tools.javac.Main
      -Dsun.boot.class.path=\j2sdk1.4.0\jre\lib\rt.jar
      \j2sdk1.4.0\lib\tools.jar

Note that this hardcodes the path to rt.jar into the javac.exe file, so this executable can only be used on the system it was created on. Note also that you cannot redistribute the javac executable because this would violate the Sun license.

Tuesday, 05 October 2004 16:17:23 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
New Snapshot

A couple of bug fixes and some cleanup, but the two major new features are the support for Whidbey (ikvmstub can now handle Whidbey's mscorlib) and the fact that getClassLoader() now returns the system class loader for statically compiled classes (instead of null).

Note that Whidbey support only means that IKVM will now run successfully on Whidbey, not that any of the new Whidbey features are available to use from Java. In particular, generic types and generic methods are not visible to Java code.

What's new:

  • Implemented support for assertions.
  • Changed TimeZone code to find an existing Java timezone that matches the current .NET TimeZone (a little better than before but it's still a very lame solution).
  • Added -r short form of -reference option to ikvmc.
  • Improved error handling for incorrect filenames/paths in ikvmc.
  • Changed TypeWrapper.GetFieldWrapper() to take a signature string instead of a TypeWrapper.
  • Statically compiled code (outside of IKVM.GNU.Classpath.dll) now returns the system class loader for Class.getClassLoader(), instead of null.
  • Changed FieldWrapper implementation to subclass based model instead of delegating to CodeEmitter instances.
  • Removed most of the CodeEmitter functionality.
  • Fixed field reflection bug (reflecting on fields in statically compiled Java types or .NET types didn't work unless you had previously reflected on the methods of the type, or dynamically compiled code against the type).
  • Added preliminary support to run on Whidbey (generic types and methods will not be visible from Java).
  • Removed map.xml support for without sig attribute.
  • Fixed reflection on .NET types to ignore methods with ByRef pointer arguments.
  • Added support to ikvmstub to skip .NET types that aren't visible from Java.

New snapshots: just the binaries and source plus binaries.

Update: I updated the snapshot to fix a regression when accessing fields where the field type class is not found.

Tuesday, 05 October 2004 13:32:00 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Monday, 27 September 2004
New Snapshot

A couple of fixes and performance enhancements for static synchronized and static native methods.

Changes:

  • Fixed ikvmc so it no longer throws an exception if no output filename is specified (or implied).
  • Fixed monitorenter instruction and synchronized methods to not throw InterruptedException.
  • Added [DebuggerStepThroughAttribute] to all methods in ByteCodeHelper that didn't have it yet.
  • Added class object caching for static synchronized and native methods.
  • Fixed Thread.stop() so that it works on suspended threads.
  • Implemented a few more random awt peer methods.
  • Removed gnu.classpath.RawData mapping from map.xml.
  • Changed ikvm.exe setting of gnu.classpath.home and java.class.path properties so that they end up in the default system properties.
  • Added IKVM.Runtime.Startup.EnterMainThread() and ExitMainThread() methods (and changed ikvm.exe) to encapsulate the setup and tear-down of the main thread.
  • Fixed JNIEnv.MonitorEnter to use ByteCodeHelper.monitorenter.
  • Fixed compiler bug that caused invoking CharSequence methods to generate a ClassCastException.
  • Changed ikvmc main stub method to call Startup.EnterMainThread() and ExitMainThread() and to pass unhandled exceptions to ThreadGroup.uncaughtException().
  • Fixed JNI method invocation to not wrap exceptions in java.lang.reflect.InvocationTargetException.
  • Fixed support for synchronized native methods.
  • Fixed bug in JNI proxy generator (only used when -Xsave option is specified).
  • Named the WaitShutdownHook and SaveAssemblyShutdownHook threads in starter.cs.
  • Fixed JNIEnv.RegisterNatives when JNI proxy generator is used.
  • Fixed Double.parseDouble to parse +Infinity correctly.

New snapshots: just the binaries and source plus binaries.

Monday, 27 September 2004 12:27:17 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 15 September 2004
New Snapshot

A new snapshot with some minor changes. In the previous entry I forgot to document two JNI restrictions:

  • JNI doesn't work with Mono 1.0 on Windows (Mono doesn't do correct P/Invoke name mangling on Windows -- this can be worked around by rebuilding IKVM.Runtime.dll with the alternative DllImport attributes in JniInterface.cs).
  • JNI libraries are never unloaded. Class loaders aren't garbage collected and consequently, native libraries aren't either.

Changes:

  • Implemented support for globbing on Windows. Both to ikvm.exe and to executables compiled with ikvmc (on ikvmc it can be disabled with the -noglobbing option).
  • Added IKVM.Runtime.Startup class that contains globbing support and a method to set Java system properties.
  • Added option to ikvmc to set Java system properties before starting the application (executables only).
  • Disabled freeing of JNIEnv unmanaged memory (for the time being) because it causes crashes on Mono (possibly due to a GC bug in Mono).
  • Added optimization to mark "side effect free" static initializers with the beforefieldinit flag. (Static compilation only.)
  • Added special case support to the verifier for the "side effect free" optimization.
  • Fixed a verifier bug that caused unverifiable code to be generated in rare circumstances (the code worked OK, it just wasn't verifiable). This was the issue that caused Xerces' org.apache.xerces.util.DOMUtil.copyInto() to fail verification.

New snapshots: just the binaries and source plus binaries.

Wednesday, 15 September 2004 15:42:50 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Thursday, 09 September 2004
JNI Code Complete & A New Snapshot

JNI is now done. Almost complete support for all J2SE 1.4 JNI functionality. There are a few (permanent) restrictions:

  • It's much slower than on a real JVM (most operations are between 10x and 100x slower).
  • You cannot create a string instance with AllocObject and then call the constructor on it with CallNonvirtualVoidMethod. This is unlikely to be a problem, because there really is no reason to go through all this trouble to create a string. The normal ways to create a strings are NewString, NewStringUTF and NewObject and these work fine. If any app were found that depended on two step string construction, a workaround for that particular app could probably be added.
  • JVM.DLL, which implements the three static invocation API functions, only works on Win32.
  • JNI_CreateJavaVM only supports the JDK 1.2 style init args and only the -D option (to define Java properties).
  • JNI_GetDefaultJavaVMInitArgs is not implemented. It's only used for JDK 1.1 style JVM creation, so I don't think it'll be used anymore. If anyone requires it, it could be implemented.
  • JNI_GetCreatedJavaVMs only returns the JavaVM if the VM was started through JNI or a JNI call that retrieves the JavaVM has already occurred.
  • JavaVM.DestroyJVM is only partially implemented (it waits until there are no more non-daemon Java threads and then returns JNI_ERR).
  • JavaVM.DetachCurrentThread doesn't release monitors held by the thread.

JVM.DLL is really cool. It's a small stub created with ILASM. It's only a few lines of code and all it really does is export three managed functions to the unmanaged world. The ability to export managed functions is is a very nice CLR feature that is relatively unknown. On a Win32 system, any native application that hosts the Sun Java Virtual Machine can now use IKVM (without modifying the native app), simply by copying the IKVM DLLs into the native applications directory and removing the real Java VM JVM.DLL from the PATH.

Changes:

  • Imported new Classpath TimeZone handling. This still leaves TimeZone support in a very broken state, but at least the Classpath VM interface is now more powerful and should hopefully allow a correct implementation when I get around to it.
  • Finally added namespaces to all public classes in IKVM.Runtime.dll. There are IKVM.Attributes, IKVM.Runtime, IKVM.Internal and IKVM.NativeCode. IKVM.Internal contains public types that really shouldn't be public, but have to be public because ikvm or ikvmc depend on them. In IKVM.NativeCode live the implementations of the GNU Classpath native methods.
    Don't use the types in the IKVM.Internal or IKVM.NativeCode namespace in your own projects, because they are likely to go away or change in incompatible ways. IKVM.Attributes and IKVM.Runtime are going to be relatively stable (but they can still change up until the 1.0 release) .
  • Added support for JNI thread detaching and waiting for all non-daemon threads to end to VMThread.java.
  • Added GetClassFromTypeHandle method to ByteCodeHelper, this replaces the direct use of NativeCode.java.lang.VMClass.getClassFromType by JNI method stubs and synchronized static methods.
  • Moved the various arraycopy methods from NativeCode.java.lang.VMSystem to ByteCodeHelper.
  • Added getSystemClassLoader method to ClassLoaderWrapper.
  • JavaException.cs: Removed message from ArrayStoreException and added InstantiationException.
  • Added invocation API support to JNI.
  • Renamed JniFrame to Frame and made it an innerclass of IKVM.Runtime.JNI.
  • Implemented more efficient way to determine active JNI method class loader in JNIEnv.FindClass.
  • Fixed JNI method resolution to look for short and long method names in each native library (previous it would first search all libraries for the short name and then in another pass would look for the long names).
  • Added support for JavaVMAttachArgs to JavaVM.AttachCurrentThread.
  • Implemented JavaVM.DestroyJavaVM.
  • Implemented JavaVM.DetachCurrentThread.
  • Implemented JavaVM.AttachCurrentThreadAsDaemon.
  • Implemented JNIEnv cleanup when thread dies.
  • Fixed JNIEnv.DefineClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.FindClass to decode name as platform specific instead of UTF-8.
  • Fixed JNIEnv.SetPendingException to do exception mapping and set the stack trace.
  • Fixed JNIEnv.ThrowNew to decode message as platform specific instead of UTF-8.
  • Fixed JNIEnv.ExceptionDescribe to clear pending exception (and to actually work).
  • Fixed JNIEnv.FatalError to decode message as platform specific instead of UTF-8.
  • Added error handling to JNIEnv.AllocObject (if you try to create interface or abstract class instance).
  • Added error handling and value type support to JNIEnv.NewObjectArray.
  • Added value type support to JNIEnv.GetObjectArrayElement and JNIEnv.SetObjectArrayElement.
  • Added error handling to the various JNIEnv.New<PrimitiveType>Array functions.
  • Added error handling to the various JNIEnv.Get<PrimitiveType>ArrayRegion and JNIEnv.Set<PrimitiveType>ArrayRegion functions.
  • Implemented growing the local ref table for native methods that use (or leak) excessive amounts of local refs.
  • Changed stack and local variable type encoding of value types and enums to System.ValueType and System.Enum respectively (instead of simply System.Object).
  • Fixed bug in bytecode compiler that caused it emit incorrect code when calling inherited methods on value types or enums.

New snapshots: just the binaries and source plus binaries.

Thursday, 09 September 2004 13:51:09 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Sunday, 05 September 2004
New Snapshot

A couple of fixes and a clean up of the JNI code. The Managed C++ JNI provider used some clever tricks and when I ported them to C# they turned into ugly hacks, so I removed them. More on that in a future entry.

Changes:

  • Changed VMAccessController and VMClassLoader to use .NET thread local storage instead of a Java ThreadLocal to reduce initialization order dependencies.
  • Added -XXsave flag to ikvm.exe to save debug image of single threaded apps that don't call System.exit().
  • Added -apartment:[sta|mta|none] flag to ikvmc (default is to apply the STAThreadAttribute to main).
  • Added Win64 support to ikvm-native.
  • Fixed class file parser to properly resize the instruction array. Previously, the compiler got confused when emitting debugging information.
  • Added code to save the assembly used for non-virtual reflective method invocations when -Xsave is used.
  • Introduced JNI type name aliases in JniInterface to make the code more readable.
  • Implemented JNI functions: EnsureLocalCapacity, PushLocalFrame, PopLocalFrame, NewWeakGlobalRef and DeleteWeakGlobalRef.
  • Fixed Debug.Assert in GhostMethodWrapper (MemberWrapper.cs).
  • Fixed JniProxyBuilder to emit ldtoken and pass it as an argument to the real JNI method.
  • Improved error handling in gnu.java.io.EncodingManager.
  • Fixed support for setting the java.library.path property on the ikvm.exe command line.
  • Fixed default java.library.path on Windows.
  • Added workaround for Mono 1.0 bug in Marshal.AllocHGlobal.
  • Fixed delegate reflection bug that caused ikvmstub not to emit the nested Method interface for delegates.
  • Added resource name mangling to make ILDASM/ILASM roundtripping work.

New snapshots: just the binaries and source plus binaries.

Sunday, 05 September 2004 11:54:45 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 31 August 2004
New Snapshot

The biggest change in this version is that the JNI implementation is now unified. It's completely rewritten in unsafe C# and a tiny bit of C (thanks to Zoltan Varga for porting the C part to Linux). This allows the same code to be used on the Microsoft CLR and on Mono. Porting the Managed C++ to C# was an interesting exercise. Because of the repetitive nature of many of the JNI functions, I really missed the C macros I used in the Managed C++ implementation, but more importantly I found that unsafe C# tries really hard to get in your way. It tries to prevent you from shooting yourself in the foot, but IMO it doesn't really succeed at that, but it does make it really hard to do clever things if you know what you're doing.

I also found some methods missing in System.Runtime.InteropServices.Marshal. There is no memcpy, no way to efficiently allocate unmanaged memory (both CoTaskAllocMem and AllocHGlobal offer functionality or guarantees that you often don't need) and there is no way to turn a delegate into an unmanaged method pointer. I had to add the following C method to convert the delegate an unmanaged method pointer:

JNIEXPORT void* JNICALL ikvm_MarshalDelegate(void* p)
{
    return p;
}

This looks like a no-op, but the P/Invoke marshaler converts the delegate to a pointer to an unmanaged thunk. Unfortunately this functionality is not exposed as a managed API.

Here's what else is new:

  • New unified JNI implementation written mostly in (unsafe) C#. The same JNI implementation now works on Mono and the Microsoft CLR and there is only a tiny bit of platform specific C code (no more Managed C++).
  • Statically compiled code that uses JNI is now portable between different platforms (obviously only if the native library is available on those platforms).
  • Implemented more JNI functionality.
  • Synchronized with GNU Classpath cvs.
  • Fixed Runtime.totalMemory() to return more than Runtime.freeMemory(). Note that the value is still pretty meaningless.
  • Added more validation to BigEndianBinaryReader to better detect invalid class files.
  • Changed ByteCode, NormalizedByteCode and ByteCodeMode enums to be byte based, for efficiency.
  • Separated wide "addressing" modes into ByteCodeModeWide to make the code simpler.
  • Added ByteCodeFlags enum to encode bytecode properties in ByteCodeMetaData.
  • Fixed ByteCodeMetaData encoding for the [ailfd]load/[ailfd]store instructions to support the wide encodings.
  • Made class file parsing eager, to prevent depending on the passed in byte array after ClassLoader.defineClass() returns.
  • Made class file parsing robust against class loaders that to attact the VM by modifying the class file byte array, while it is being parsed.
  • Made many Optimizations to class file parsing.
  • Added more validation to class file parsing.
  • Fixed class file parsing to throw a ClassFormatError when a class loader is trying to redefine java.lang.Object.
  • Implemented class loader namespace support for native libraries.
  • Fixed FieldWrapper.Link() to rethrow exception if linking fails (instead of silently swallowing it).
  • Made the __ static initialization trigger field protected instead of public.
  • Use a more efficient CIL sequence for instanceof instruction.
  • Fixed stupid bug in Double/Float.parse(). It used the local sensitive version of System.Double.Parse().
  • Fixed java.io.System.setErr().

New snapshots: just the binaries and source plus binaries.

Update: I forgot to thank Zoltan Varga for porting the platform specific part of the new JNI implementation to Linux.

Tuesday, 31 August 2004 13:55:43 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 20 August 2004
Non-virtual Method Invocation

In Tuesday's snapshot I finally implemented the CallNonvirtualMethod JNI routines. I've never seen any JNI code that actually uses them, but I obviously needed to implement them at some point.

For those of you unfamiliar with these routines, let me explain them briefly. Normally when you call a virtual method, the runtime type of the instance the method is called on determines the actual method you end up at (so called virtual method dispatch), but these routines allow you to call a base class method as well. So, if you'd use CallNonvirtualObjectMethod on java.lang.Object.toString() on a java.lang.String instance (let's say “foo“), the result you'd get would be java.lang.String@17e809 instead of the string “foo“.

The CLR Reflection API doesn't have the ability to invoke a virtual method non-virtually. Unlike Java however, the CLR does allow the CIL call instruction to be used to call arbitrary methods non-virtually (the Java invokespecial method can only be used to call methods in the same class or in a base class). So the obvious way to implement these routines, would seem to be to dynamically emit some code that calls the requested method non-virtually, but there is problem with this approach, the JNI routines don't enforce accessibility (they allow you to call non-public methods) and the CLR enforces accessibility. I got around this by using RuntimeMethodHandle.GetFunctionPointer() incombination with the CIL calli instruction. This also has the advantage that the dynamically emitted stubs can be reused for methods with the same signature, because the target address isn't baked into the method, but passed in.

Non-orthogonality

While I don't have a very compelling use case for non-virtual method invocation through reflection, based on API orthogonality and completeness a case could be made for it, but more importantly, I think that ECMA/Microsoft should consider changing the CLI spec to allow non-verifiable code to bypass the accessibility checks. Note that the new DynamicMethod in Whidbey already allows accessibility checks to be bypassed (and note also that whoever wrote the doc for DynamicMethod doesn't understand the difference between visibility and accessibility).

The trick of using calli to bypass the accessibility checks doesn't work for virtual method invocation, because there is no way to dynamically and efficiently get the function pointer of a method by applying the virtual method dispatch rules (i.e. there is no dynamic equivalent of the CIL ldvirtftn instruction).

To summarize, here is what I think should be changed:

  • CLR shouldn't bother enforcing accessibility in non-verifiable code.
  • MethodInfo should get a new InvokeNonvirtual method.
  • MethodInfo should get a GetFunctionPointer(object) method that returns the method pointer that results from virtual dispatch.
Friday, 20 August 2004 09:11:25 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]