# Wednesday, 01 August 2007
New Hybrid Snapshot

Major code bloat. I've now integrated the bulk of the OpenJDK code that isn't awt or swing. As a result the IKVM.Hybrid.GNU.Classpath.OpenJDK.dll assembly size has grown to about 26 MB. API coverage compared with JDK 1.6 is now at more than 98%. Note that doesn't mean that everything will work, because some back-end implementations are stubbed out or not included.

Disclaimers apply. I haven't done a full test pass on this build.

Changes:

  • OpenJDK: Integrated javax.management package (and sub packages).
  • OpenJDK: Integrated java.lang.management package (only a stub back-end implementation though).
  • OpenJDK: Integrated javax.imageio package (excluding the jpeg support, because OpenJDK uses native code for that).
  • OpenJDK: Integrated javax.activation, javax.annotation, javax.jws, javax.lang.model, javax.tools, javax.xml.*, org.jcp.xml.dsig.internal, org.relaxng.datatype, org.w3c.dom.*, org.xml.sax.* packages.
  • OpenJDK: Integrated javax.sql.* packages.
  • OpenJDK: Integrated javax.accessibility, javax.transaction, javax.activity packages.
  • OpenJDK: Integrated javax.print.* packages (no back-end implementation and ServiceUI is stubbed.)
  • OpenJDK: Integrated org.omg.*, javax.rmi.*, javax.sound.*, org.ietfs.jgss packages.
  • Fixed JNI NewObject method to actually create an object of the requested class, instead of the class of the constructor.
  • Added method name clash handling for AOT access stub methods.

Binaries available here: ikvmbin-hybrid-0.35.2769.zip.

Wednesday, 01 August 2007 10:16:12 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Thursday, 26 July 2007
New Hybrid Snapshot

A major step in integrating OpenJDK code. One of the bottlenecks was the fact that currently OpenJDK is missing most of the crypto code, but thanks to the IcedTea project I've been able to integrate the OpenJDK java.security package and packages that depend on the crypto code (like java.net). I've also rewritten all socket implementation classes (both classic and nio) based on the OpenJDK code. FileChannelImpl and the direct and mapped byte buffers still need to be converted.

Some of the integrated packages don't have any back-end implementation (e.g. smartcardio and jgss). I'm not likely to implement smartcard support and will revisit the security and crypto stuff once Sun releases the crypto code.

Disclaimers apply. I haven't done a full test pass on this build.

Changes:

  • OpenJDK: Added support for "loading" fake native libraries from VFS and removed hack to bypass loadLibrary() call in System.initializeSystemClass().
  • OpenJDK: Integrated OpenJDK packages: java.net, java.security, java.util.jar, javax.naming, javax.net, javax.security, javax.smartcardio, java.nio.charset, java.nio.channels, java.nio.channels.spi
  • OpenJDK: Integrated IcedTea crypto/security classes.
  • OpenJDK: Fixed a race condition in Thread.interrupt().
  • Allow Object[] to be cast/assigned to ghost array. Fix for bug 1757889.
  • Fixed assembly annotation support (bug 1721688).
  • Added ikvmc warning when annotation type isn't found.
  • Added WINDOWS constant to ikvm.internal.Util to check if we're running on Windows.

Binaries available here: ikvmbin-hybrid-0.35.2763.zip.

Thursday, 26 July 2007 14:15:37 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Wednesday, 11 July 2007
.NET Framework Security Update

Yesterday Microsoft released a security update for the .NET Framework that fixes the bug I reported last December. I'll write a more detailed analysis (including a proof-of-concept exploit) in a couple of weeks.

Wednesday, 11 July 2007 07:10:17 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Wednesday, 04 July 2007
New Hybrid Snapshot

I decided to take a break from integrating new OpenJDK packages and instead focus on running some test cases and work on stabilization. Contrary to the previous hybrid snapshots, this version should be fairly usable. Feedback is appreciated.

Changes:

  • OpenJDK: Implemented java.util.concurrent.locks.LockSupport.
  • OpenJDK: Fixed race condition in Thread.interrupt() that could cause cli.System.Threading.ThreadInterruptedException to be thrown from interruptable waits/sleep.
  • OpenJDK: Imported and modified java.util.concurrent.locks.AbstractQueuedSynchronizer to make it more efficient and to remove the use of ReflectionFactory & Unsafe to reduce initialization order dependencies.
  • OpenJDK: Changed unsafe to use more efficient internal helper class to copy java.lang.reflect.Field and make it accessible (this also reduces initialization order dependencies).
  • OpenJDK: Added lib/logging.properties to VFS and implemented an additional VFS operation required for reading it.
  • OpenJDK: Changed VFS ZipEntryStream.Read() to always try to read the requested number of bytes, instead of returning earlier. For maximum compatibility with real file i/o.
  • OpenJDK: Commented out system property setting in sun.misc.Version that was setting some version properties to bogus values.
  • OpenJDK: Fixed ObjectInputStream.latestUserDefinedLoader() to skip mscorlib stack frames.
  • OpenJDK: Added support to VFS for the VFS root directory.
  • OpenJDK: Fixed FieldAccessorImpl to check the type of the object passed in.
  • OpenJDK: "Implemented" ClassLoader.retrieveDirectives() by returning empty AssertionStatusDirectives object. Enabling assertions on the ikvm.exe command line is still not implemented.
  • OpenJDK: Switched to javac compiler for building OpenJDK sources.
  • Added workaround for .NET 1.1 reflection bug that causes methods that explicitly override a method to show up twice.
  • Changed handling of ldfld & lfsfld opcodes in remapper to bypass "magic".
  • Restructured handling of fields defined in map.xml to enable referencing them from map.xml method bodies.
  • Fixed java.security.VMAccessController to make sure that if there is only system code on the stack, the resulting AccessControlContext isn't empty.
  • Updated to compile with GNU Classpath HEAD.

Binaries available here: ikvmbin-hybrid-0.35.2741.zip.

Wednesday, 04 July 2007 15:04:10 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 27 June 2007
New Hybrid Snapshot

I'm getting tired of writing the disclaimers and warnings, but they still apply.

The most important change in this snapshot is that there's now a virtual file system for the java.home directory. This has been a long time coming, but the proverbial straw was the fact that the OpenJDK timezone code reads files from the java.home/lib/zi/ directory (IMHO they really should be using resources for these things).

Currently the virtual java.home directory is C:\.virtual-ikvm-home\ on Windows and /.virtual-ikvm-home/ on Unix, but this is subject to change (please let me know if you have thoughts on this). The only contents in there so far is the /lib/zi/ directory tree and only a few file operations are supported (notably the ones required by the timezone code), but expect that eventually all (read-only) file system operations will be supported and more virtual files to appear in there.

Why a Virtual File System Instead Of a Real Java Home Directory?

The main reason is that I want IKVM to behave like a .NET library as much as possible. That means it should be possible to install it into the GAC and support the versioning and side-by-side capabilities of .NET, that's very hard to do when you have to manage real directories.

Changes:

  • OpenJDK: Integrated java.util.spi, java.util.prefs and java.util.logging packages.
  • OpenJDK: Integrated java.text and java.text.spi packages (except for java.text.Bidi class, for which Sun uses native code, so we'll continue to use GNU Classpath's pure Java version.)
  • OpenJDK: Changed build script to include all resources from OpenJDK generated resources.jar.
  • OpenJDK: Integrated java.rmi package.
  • OpenJDK: Changed system/extension class loader creation to make sure that an extension class loader always exists if there is a non-assembly system class loader.
  • OpenJDK: Improved exception handling in java.io.FileDescriptor.
  • OpenJDK: Removed AccessController.doPrivileged() call in Unsafe.fieldOffset(), to work around Mauve brokenness.
  • OpenJDK: Implemented the beginnings of a virtual file system for the java.home directory.
  • Changed JVM.IsUnix to use Environment.OSVersion.Platform.

Binaries available here: ikvmbin-hybrid-0.35.2734.zip.

Wednesday, 27 June 2007 08:57:35 (W. Europe Daylight Time, UTC+02:00)  #    Comments [3]
# Thursday, 21 June 2007
New Hybrid Snapshot

Another hybrid snapshot update. Just a reminder again: These snapshots have not been tested extensively and are known to be broken (and are much more broken than the non-hybrid snapshots I used to release). They are only intended to be used for testing and getting a feel of how things are going. If you find a bug specific to this snapshot, please don't file a bug on SourceForge, simply send a message to the ikvm-developers list or to me directly.

I'm not going to list everything that's known to be broken, but I will say that Eclipse 3.2 still runs, so at least some parts do work ;-)

This build also includes several GNU Classpath fixes that I haven't yet checked in, so if you're trying to do a hybrid build from cvs you'll end up with something even more broken than this build. [Update: I checked them in.]

Changes:

  • OpenJDK: Upgraded to OpenJDK bundle b13.
  • OpenJDK: Added more resources.
  • OpenJDK: Integrated java.lang.annotation and java.lang.ref packages.
  • OpenJDK: Integrated java.io and java.util packages.
  • Added -serialver option to ikvmstub that forces stub generator to include serialVersionUID field for all serializable classes (to make Japi results more accurate).
  • Fixed GetParameterAnnotations() to return the correct array length for instancehelper__ methods (static methods that represent instance methods on remapped types).
  • Fixed ikvm.io.InputStreamWrapper.available() to return non-zero when more data is available (as suggested by Mark Reinhold). Removed the NormalizerDataReader specific fix from map.xml.
  • Fixed ikvmstub to better handle private interface implementations. (Fixes System.Web.UI.Control subclassing issue).

Binaries available here: ikvmbin-hybrid-0.35.2728.zip.

Thursday, 21 June 2007 07:23:42 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Tuesday, 19 June 2007
Five Years

Wow. Today it's five years ago that I started blogging about IKVM. I'd write more, but I'm having too much fun working on integrating the OpenJDK libraries :-)

Tuesday, 19 June 2007 06:32:18 (W. Europe Daylight Time, UTC+02:00)  #    Comments [3]
# Thursday, 14 June 2007
First OpenJDK Bug

Yesterday I encountered the first OpenJDK "bug". Sun uses IBM's ICU library for Unicode and Globalization support and it does a couple of DataInputStream.read(byte[]) calls without checking the return value. On IKVM the InputStream that ClassLoader.getSystemResourceAsStream() returns is an ikvm.io.InputStreamWrapper and that always returns 0 from available() when it is wrapping a non-seekable stream and the LZInputStream that decompresses compressed resources is a non-seekable stream. The reason this is significant is that ICU wraps a BufferedInputStream around the InputStreamWrapper and BufferedInputStream uses available() to figure out if it should read more data from the underlying stream if there's still room in the byte array passed in to read(byte[]) after the data from the buffer has been copied. So as long as the underlying stream has an implementation of available() that never returns 0, ICU works fine. I filed a bug report.

Issues like this demonstrate why cloning a large platform is so incredibly difficult. Real world code is bound to have dependencies on implementation details like this. So simply reimplementing a spec isn't sufficient.

For the time being I've added a workaround to map.xml (using the new <replace-method-call /> support) to modify the ICU method to call readFully(byte[]) instead of read(byte[]), but I'm considering rewriting the resource decompression to have available() return the number of bytes still available to read from the stream. There may be other code out there that depends on it.

Thursday, 14 June 2007 12:26:19 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Wednesday, 13 June 2007
New Hybrid Snapshot

A new snapshot. I integrated all classes in the java.lang package (but not the sub-packages) and this means I have now resolved most VM/Library initialization issues that I was a bit worried about before embarking on this. I had to include one new hackish (or AOP if you will) feature in map.xml: The ability to replace method invocations with arbitrary CIL code sequences. This was in particular necessary to "comment out" the loadLibrary("zip") call in System.initializeSystemClass(). Here's what that looks like in map.xml:

<class name="java.lang.System">
  <method name="initializeSystemClass" sig="()V">
    <replace-method-call class="java.lang.System" name="loadLibrary" sig="(Ljava.lang.String;)V">
      <code>
        <pop />
      </code>
    </replace-method-call>
  </method>
</class>

This simply means that all System.loadLibrary() invocations in System.initializeClass() will be replaced with a pop instruction (to discard the string argument).

Changes:

  • OpenJDK: Integrated java.lang package..
  • OpenJDK: Integrated java.util.regex package.
  • OpenJDK: Integrated java.text.Normalizer and support classes.
  • OpenJDK: New StringHelper.java based on OpenJDK's String.java.
  • OpenJDK: Implemented using the entry assembly's class loader as system class loader.
  • OpenJDK: Replaced zip library loading hack with "replace-method-call" hack.
  • OpenJDK: Implemented the hooks to set the system class loader to the entry assembly's class loader if java.class.path and java.ext.dirs properties aren't set.
  • OpenJDK: Various fixes.
  • Added leave CIL opcode support to remapper.
  • Added support for replacing constructor and static initializer method bodies in map.xml.
  • Added support for locally (i.e. per method) replacing method calls in map.xml.
  • Added optimization to ikvmc for large string literals that are only used to call .toCharArray() on.

Binaries available here: ikvmbin-hybrid-0.35.2720.zip.

Wednesday, 13 June 2007 08:35:28 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 12 June 2007
Array Initialization

One of the many limitations of the Java class file format is that it doesn't support an efficient way of initializing an array. Take the following class for example:

class Foo
{
  private static final int[] values = { 1, 2, 3, 4 };
}

This looks harmless enough, but it is compiled almost exactly the same as this:

class Foo
{
  private static final int[] values;

  static
  {
    int[] temp = new int[4];
    temp[0] = 1;
    temp[1] = 2;
    temp[2] = 3;
    temp[3] = 4;
    values = temp;
  }
}

If you have an array with a few thousand entries, this is starting to look pretty inefficient. Here's how C# compiles the equivalent code (but with an array with more elements, for small arrays the C# compiler does the same as the javac):

using System.Runtime.CompilerServices;

class
Foo
{
  private static readonly int[] values;

  static Foo()
  {
    int[] temp = new int[128];
    RuntimeHelpers.InitializeArray(temp, LDTOKEN($$field-0))
    values = temp;
  }
}

This is pseudo C#, because there's no C# language construct that allows you to directly load a field's RuntimeFieldHandle as the ldtoken CIL instruction does. What happens here is that the C# compiler emits a global read-only data field and then uses RuntimeHelpers.InitializeArray() to copy this field into the array. If you're on a little endian machine, this is simply a memcpy (after checking that the sizes match).

On occasion I've toyed with the idea of making ikvmc recognize array initialization sequences and then converting them into this more efficient form. The reason that I never got around to it is that many people know that array initialization is inefficient and use a workaround: String literals.

Here's a fragment from OpenJDK's java.lang.CharacterData00 class:

static final char X[] = (
  "\000\020\040\060\100\120\140\160...
  ...
  ...
  ...\u1110\u1120\u1130").toCharArray();

I've omitted most of the data, but the string has 2048 characters (and it's not the only one in this class). String literals can be efficiently stored in the class file format, so this is more efficient than explicitly initializing the array elements.

However, on IKVM there's a downside to this approach. Java requires that all string literals are interned and on .NET the string intern table keeps strong references to the interned strings (the Sun JVM uses weak references to keep track of interned strings). So that means that the 2048 character string above that is only used once will stay in memory for the entire lifetime of the application.

Fortunately, this particular scenario is easily detected in the bytecode compiler and can be optimized to use RuntimeHelper.InitializerArray().

Unfortunately, it turns out that the Reflection.Emit API that takes care of creating these read-only global data fields is a little too helpful and hence broken. What I didn't mention earlier is that these read-only global data fields need to have a type and that type has to be an explicit layout value type of the right size. ModuleBuilder.DefineInitializedData() automatically defines this type, but instead of making it private it creates a public type.

The type created by ModuleBuilder.DefineInitializedData() is named $ArrayType$<n> where <n> is the size of the value in bytes. The size is encoded in the type name to make it easy to reuse the same type for other fields that have the same size. It turns out we can abuse this knowledge to work around the limitations of ModuleBuilder.DefineInitializedData() by pre-creating the value type with the right name. Now we can have full control over the properties of the generated type (as long as it's compatible with the requirements of the read-only global field, of course).

Final note: While writing this blog entry I found that Mono's implementation of RuntimeHelpers.InitializeArray() leaves something to be desired.

Tuesday, 12 June 2007 08:45:48 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]