# Monday, 14 November 2011
IKVM.NET 7.0 Release Candidate 0

The first release candidate is available. No changes (except the version number and strong naming) relative to the last development snapshot.

What's New (relative to IKVM.NET 0.46):

  • Integrated OpenJDK 7 b147.
  • Implemented Java 7 JVM changes.
  • Java annotations on statically compiled code are now returned as java.lang.reflect.Proxy objects for compability with broken code that assumes this is always the case.
  • Added delegate conversion for java.lang.reflect.InvocationHandler to ikvm.runtime.Delegates.
  • Various remap file improvements.
  • Changed build and JNI code to use different names for the Windows x86 and x64 versions of the native dll.
  • Retain reflection field ordering for ikvmc compiled code (not required by spec, but to improve compatibility with broken code).
  • Various AWT fixes.
  • Interop between java.lang.AutoCloseable and System.IDisposable.
  • Various build system improvements.
  • Added ikvmc -warnaserror option.
  • Fixed java.io.FileOutputStream in append mode to use atomic append.
  • Various performance improvements.
  • Added -Xnoglobbing option to ikvm.exe.
  • Various minor fixes.
  • Implemented dual stack sockets (Windows Vista and up only).
  • Implemented platform MBean server.
  • Implemented SocketChannel.sendOutOfBandData().
  • Implemented DatagramChannel multicast methods.
  • Removed mapping of System.Security.VerificationException to java.lang.VerifyError.
  • IKVM.Reflection: Massive enhancements to support a very large portion of the managed PE capabilities (much more than System.Reflection.Emit).

When the final release is done, it will include the full release notes.

Binaries available here: ikvmbin-7.0.4335.0.zip

Sources: ikvmsrc-7.0.4335.0.zip, openjdk7-b147-stripped.zip

Monday, 14 November 2011 14:01:32 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
Managed PE File Types

Just a quick blog about the different types of managed PE files.

Here's a table:

Description C# compiler switch PE type machine corflags
MSIL /platform:anycpu (default) PE32 x86 ILONLY
MSIL 32 bit preferred /platform:anycpu32bitpreferred PE32 x86 ILONLY | 32BITREQUIRED | 32BITPREFERRED
x86 managed /platform:x86 PE32 x86 ILONLY | 32BITREQUIRED
x86 mixed n/a PE32 x86 32BITREQUIRED
x64 managed /platform:x64 PE32+ x64 ILONLY
x64 mixed n/a PE32+ x64  
ARM managed /platform:arm PE32 ARM ILONLY
ARM mixed n/a PE32 ARM  

The MSIL 32 bit preferred and ARM types are new in .NET 4.5. I've left out Itanium, because it's not that interesting.

For legacy reasons pure MSIL files specify x86 as the machine type and when you really want to target just x86 you need to set the (confusingly named) COMIMAGE_FLAGS_32BITREQUIRED flag.

When the COMIMAGE_FLAGS_32BITREQUIRED flag was introduced (back in .NET 1.0) they probably intended it to just mean that the image required a 32 bit architecture (not necessarily x86), but then the C# compiler mapped the /platform:x86 switch to this flag and they were locked into it meaning just x86.

Starting with Visual Studio 2010, the default platform target project setting for executables changed from Any CPU to x86. This was probably because x64 systems were becoming more popular and more people started running into the poor x64 JIT performance and its many bugs.

With the introduction of ARM support in Windows 8 there was now a need for a new flag to avoid the x64 JIT, but still be cross platform, so the COMIMAGE_FLAGS_32BITPREFERRED flag was invented. For downward compatibility, this flag is not set by itself, but instead modifies the meaning of the COMIMAGE_FLAGS_32BITREQUIRED flag, to allow .NET 4.0 systems to continue to run the executable in 32 bit mode (they simply ignore the new flag) while on ARM systems the system can see that the executable is not x86 specific because the new flag is set.

Finally, when your PE file contains unmanaged exports the ILONLY flag must not be set or the loader will refuse to load the file.

Monday, 14 November 2011 09:18:51 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Monday, 07 November 2011
New Development Snapshot

I've implemented the JDK 7 method overriding behavior. So this should be the final snapshot before the 7.0 release candidate.

It's really a testament to the .NET method overriding model that I was able to easily implement all the weird JDK 7 behavior without resorting to any hacks. It's a simple matter of using MethodAttributes.NewSlot and TypeBuilder.DefineMethodOverride() in the right places.

Changes:

  • Updated LICENSE and THIRD_PARTY_README to OpenJDK 7. Thanks to Martin for pointing out in the comments this was still missing.
  • Implemented JDK 7 method overriding behavior (minus the part 5 bug).

Binaries available here: ikvmbin-7.0.4328.zip

Monday, 07 November 2011 07:46:12 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
Java Method Overriding Is FUBAR Part 6 of ∞

Parts 1 through 5 described behavior that is new in JDK 7, but there are also pre-existing bugs.

When you declare a static method as final (which makes no sense and there is no good reason this was ever allowed) you disallow a virtual method in a base class from being overridden:

public class A
{
  public void foo() {
    System.out.println("A.foo");
  }
}

public class B extends A
{
  public static final void foo() {
    System.out.println("B.foo");
  }
}

public class C extends B
{
  public void foo() {
    System.out.println("C.foo");
  }
}

public class test
{
  public static void main(String[] args) {
    new C();
  }
}

After you compile this (the easiest trick is probably to rename the B.foo method to f__ and after you compile use a hex editor to patch the resulting class file) and run it, you get:

Exception in thread "main" java.lang.VerifyError: class C overrides final method foo.()V
        at ...

Like I said, this is not new. I also tried this on JDK 1.1 and JDK 1.5 and on 1.1 behaves as expected and 1.5 has the modern behavior.

Monday, 07 November 2011 07:12:25 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Friday, 04 November 2011
Java Method Overriding Is FUBAR Part 5 of ∞

One of the bizarre things is that the method overriding behavior changed between Java 6 and Java 7 (and HotSpot preserves the Java 6 behavior for classes that have major class file version < 51), but the spec doesn't mention this at all, where it generally is pretty good about mentioning different behavior based on class file version.

Here's an example that uses a class hierarchy that mixes version 50 and 51 classes to confuse HotSpot JDK 7:

package p1;

public class A
{
  { foo(); }
  public void foo() {
    System.out.println("A.foo");
  }
  public static void main(String[] args) {
    new p4.D();
  }
}

package p2;

public class B extends p1.A
{
  { foo(); }
  void foo() {
    System.out.println("B.foo");
  }
}

package p3;

public class C extends p2.B
{
  { foo(); }
  void foo() {
    System.out.println("C.foo");
  }
}

package p4;

public class D extends p3.C
{
  { foo(); }
  void foo() {
    System.out.println("D.foo");
  }
}

Here's a table view of the structure:

p1   p2   p3   p4
A      
  B    
    C  
      D

After you compile this with javac 7 (again using the trick to first compile without the public modifier on A.foo) then use a hex editor to modify p3/C.class to change the class file version from 51 (33 hex) to 50 (32 hex) at offset 7 in the file.

When you run it you get:

D.foo
D.foo
Exception in thread "main" java.lang.AbstractMethodError: p3.C.foo()V
        at p3.C.(C.java:5)
        at p4.D.(D.java:3)
        at p1.A.main(A.java:10)

(This is a variation of the bug I reported here.)

Friday, 04 November 2011 08:54:44 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
Java Method Overriding Is FUBAR Part 4 of ∞

Another example of broken JDK 7 behavior:

package p1;

public class A
{
  { foo(); }
  void foo() {
    System.out.println("A.foo");
  }
  public static void main(String[] args) {
    new p3.F();
  }
}

package p2;

public class B extends p1.A
{
  { foo(); }
  void foo() {
    System.out.println("B.foo");
  }
}

package p2;

public class C extends p2.B
{
  { foo(); }
  void foo() {
    System.out.println("C.foo");
  }
}

package p3;

public class D extends p2.C
{
  { foo(); }
  void foo() {
    System.out.println("D.foo");
  }
}

package p1;

public class E extends p3.D
{
 { foo(); }
 void foo() {
   System.out.println("E.foo");
 }
}

package p3;

public class F extends p1.E
{
  { foo(); }
  void foo() {
    System.out.println("F.foo");
  }
}

To make this is a little clearer, here's a table view of the structure:

p1   p2   p3  
A    
  B  
  C  
    D
E    
    F

When you run it you get:

F.foo
C.foo
C.foo
F.foo
F.foo
F.foo

Not only is this not compliant with the spec, it just plain makes no sense.

Friday, 04 November 2011 08:32:31 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
Java Method Overriding Is FUBAR Part 3 of ∞

For another example of weird JDK 7 behavior, start with the code from part 2 and make B.foo final.

When you run that you get:

C.foo
B.foo

So now C.foo suddenly doesn't override B.foo. This also violates the spec, because when a method that would have been overridden is marked final the verifier should fail to load the class. This shows that the verifier and vtable layout code have different ideas about method overriding semantics.

Friday, 04 November 2011 07:33:47 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
Java Method Overriding Is FUBAR Part 2 of ∞

In part 1 of this series I argued that the spec is broken, now I'll show the first example where the reference implementation does not implement the spec.

package p1;

public class A
{
  { foo(); }
  public void foo() {
    System.out.println("A.foo");
  }
  public static void main(String[] args) {
    new p3.C();
  }
}

package p2;

public class B extends p1.A
{
  { foo(); }
  void foo() {
    System.out.println("B.foo");
  }
}

package p3;

public class C extends p2.B
{
  void foo() {
    System.out.println("C.foo");
  }
}

(If you want to compile this with javac you'll need to first compile it with a version of A that does not have a public foo method and then change A and just recompile A.)

Now the question is does C.foo override B.foo? According to the spec it does not, but when you compile this with javac from JDK 7 (because the class file version needs to be 51 to get the new behavior) and run it you get:

C.foo
C.foo

So C.foo overrides both A.foo and B.foo.

Friday, 04 November 2011 07:27:31 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
Java Method Overriding Is FUBAR Part 1 of ∞

The Java Virtual Machine Specification Java SE 7 Edition finally has a section about method overriding (5.4.5). Unfortunately, it does not document the behavior of the Java 7 reference implementation, nor the Java 6 behavior. It also turns out to be a bad design.

Suppose you have a third party widget framework that has a class:

package org.example;

public class Widget
{
}

And your application has a subclass of Widget:

package fubar.tech;

public class MyWidget extends org.example.Widget
{
  void frob() { ... }
}

The frob() method is package private and you don't want external code calling it. Now a new version of the widget framework is released that adds a frob method to Widget:

package org.example;

public class Widget
{
  public void frob() { }
}

Starting with Java 7 (and this particular behavior is defined by the spec) your package private MyWidget frob() will now override Widget.frob() and hence be callable by anyone who happens to have a reference to your object. If MyWidget.frob() does anything security critical, the third party framework has now introduced a vulnerability in your code.

Java already had a poor story for adding virtual methods to public classes, but this change has made it even worse.

Friday, 04 November 2011 07:01:39 (W. Europe Standard Time, UTC+01:00)  #    Comments [7]
# Monday, 31 October 2011
New Development Snapshot

Fixed the remaining known OpenJDK 7 issues, apart from method overriding, which finally made it into the JVM spec (section 5.4.5), but it's not clear yet what the actual behavior should be. It's a little ridiculous, but almost 16 years after the release of Java 1.0 a fundamental part of the JVM is still poorly documented and implemented.

Changes:

  • Added ikvmc -platform:arm and -platform:anycpu32bitpreferred options.
  • Bug fix. Make sure sun.misc.Launcher is initialized before setting a security manager, because Launcher assumes there is no security manager yet.
  • Added ikvmstub -forwarders option to support ".NET Core profile" assemblies.
  • Implemented com.sun.security.auth.module.NTSystem.getCurrent().
  • Fixed Throwable.fillInStackTrace() to respect non-writeable stack trace.
  • Fixed Throwable deserialization issues introduced with OpenJDK 7.
  • Fixed Throwable.getSuppressed() to return a global empty array (OpenJDK tests depend on that).
  • Fixed java.lang.ProcessImpl pipe to flush writes.
  • Fixed java.lang.ProcessImpl.openForAtomicAppend() to actually use atomic append.
  • Fixed java.lang.ProcessImpl to clear out the existing environment variables when an envblock is specified.
  • Implemented java.lang.ProcessImpl redirectErrorStream for case where stdout is not also redirected.
  • Fixed core library build issue. java.lang.AutoCloseable.close() method may be used before it has been linked.
  • Bug fix. When calling a final method on a remapped type we can't call the instance method in the remapped type, but we have to call the instancehelper instead.
  • Starting with class file version 51 the <clinit> methods need to be static.
  • Added java.nio.file security manager checks.
  • Added workaround for another x64 JIT bug.
  • IKVM.Reflection: Added .NET 4.5 metadata enum values.
  • IKVM.Reflection: Added support for PortableExecutableKinds.Preferred32Bit.
  • IKVM.Reflection: Added support for process architecture in assembly flags.

Binaries available here: ikvmbin-7.0.4321.zip

Monday, 31 October 2011 12:01:05 (W. Europe Standard Time, UTC+01:00)  #    Comments [4]