# Monday, 30 November 2009
IKVM 0.42 Release Candidate 3

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.42.0.3.
  • Fix for bug #2887316.
  • Eyal Alaluf fixed a massive performance bug that caused IKVM.OpenJDK.Core.dll to be 1.4 MB too big and allocate a couple of MB of unnecessary strings on initialization.
  • Fixed JNI to use system class loader instead of bootstrap class loader when no Java code is on the stack.
  • Fixed JNI method lookup algorithm.

Binaries available here: ikvmbin-0.42.0.3.zip

Sources: ikvm-0.42.0.3.zip, openjdk6-b16-stripped.zip

Monday, 30 November 2009 07:00:02 (W. Europe Standard Time, UTC+01:00)  #    Comments [4]
# Wednesday, 25 November 2009
Running Eclipse with NGEN

In the weeks before PDC I've been working on compiling Eclipse with ikvmc. This works was triggered by Mainsoft's Eyal Alaluf who asked me to work on this and also provided a desperately needed starting point. I had wanted to do this for ages, but didn't feel like struggling with the Eclipse build system to figure out how to get started.

A couple of the changes in the most recent development snapshot are specifically related to this. In particular the ability for custom assembly class loaders to be called when the module initializer is run. This enables the statically compiled Eclipse OSGi bundles to be lazily activated on first use.

Instructions

Here are the steps needed to compile Eclipse 3.4.2 x86 on Windows:

  1. Download eclipse-SDK-3.4.2-win32.zip
  2. Download ikvmbin-0.43.3595.zip
  3. Download ikvm-eclipse-0.1.zip
  4. Unzip eclipse-SDK-3.4.2-win32.zip
  5. Open a Command Prompt in the just unzipped eclipse directory
  6. Unzip ikvmbin-0.43.3595.zip in that directory
  7. Unzip ikvm-eclipse-0.1.zip in that directory
  8. Create a directory for the compiled plugins:
    md plugins-compiled
  9. Run ikvmc to compile the eclipse plugins:
    ikvm\bin\ikvmc @response0.txt
    ikvm\bin\ikvmc @response1.txt
    (Ignore the warnings and note that this takes a while and requires a lot of memory. I haven't tested this on a 32 bit machine, it may well run out of address space there.)
  10. You can now run "eclipse-clr.exe" to start Eclipse. Note that if you compare startup times, the first time that Eclipse starts it does some initial configuration, so don't compare the first startup with the subsequent ones.
  11. Optionally you can run ngen-all.bat to compile all assemblies to native code. Make sure that you have the x86 version of ngen.exe in your path. Note that this also takes a while.

Source Code

The sources for eclipse-clr.exe are in this Visual Studio 2008 solution. It's pretty small and most of what it does is configure and hook OSGi to change the bundle loading and initialization. If you want to build eclipse-clr.exe, you first have to run ikvmc on response0.txt, then build eclipse-clr.exe (it depends on the OSGi assembly built with response0.txt) and after that you can run ikvmc on response1.txt (it depends on eclipse-clr.exe, because that contains the custom assembly class loader used for the bundles).

The response0.txt and response1.txt files were generated from the OSGi manifests and if there is interest I can publish the source to that as well, but is pretty hacky.

Performance

When compiled to native with ngen, Eclipse starts up faster than with JDK 1.6 on my systems. In theory the private working set should also be significantly less, allowing multiple Eclipse instances to use far less memory.

Disclaimer

This is just a technology demonstration, not production code and has not been extensively tested.

Wednesday, 25 November 2009 06:32:30 (W. Europe Standard Time, UTC+01:00)  #    Comments [20]
# Tuesday, 10 November 2009
Going Crazy with Generics, or The Story of ThreadLocal(Of T)

Jon Skeet recently blogged about the performance of [ThreadStatic] versus the new .NET 4.0 ThreadLocal<T>. I was surprised to see that ThreadLocal<T> was faster than [ThreadStatic], because ThreadLocal<T> uses [ThreadStatic] as the underlying primitive.

How do you go from a static field to a per instance field? It's simple, once you think of it. You (ab)use generic types. Here's a simplified ThreadLocal<T>:

public class ThreadLocal<T>
{
  HolderBase holder;
  static int count;
  static Type[] types = new Type[] { typeof(C1), typeof(C2), typeof(C3) };

  abstract class HolderBase
  {
    internal abstract T Value { get; set; }
  }

  class C1 { }
  class C2 { }
  class C3 { }

  class Holder : HolderBase
  {
    [ThreadStatic]
    static T val;

    internal override T Value
    {
      get { return val; }
      set { val = value; }
    }
  }

  public ThreadLocal()
  {
    holder = MakeHolder(Interlocked.Increment(ref count) - 1);
  }

  HolderBase MakeHolder(int index)
  {
    Type t1 = types[index % 3];
    Type t2 = types[(index / 3) % 3];
    Type t3 = types[index / 9];
    Type type = typeof(Holder<,,>).MakeGenericType(typeof(T), t1, t2, t3);
    return (HolderBase)Activator.CreateInstance(type);
  }

  public T Value
  {
    get { return holder.Value; }
    set { holder.Value = value; }
  }
}

The real ThreadLocal<T> type in .NET 4.0 beta 2 is much more complex, because it has to deal with recycling the types and protecting against returning a value from a recycled type. It also uses a higher base counting system  to number the types, the maximum number of types generated (per T) is 4096 in beta 2. After you allocate more than that, it falls back to using a holder type that uses Thread.SetData().

I'm not sure what to make of this. It's a clever trick, but I think it ultimately is too clever. I benchmarked a simpler approach using arrays (where each ThreadLocal<T> simply allocated an index in the [ThreadStatic] array) and it was a little bit faster and doesn't suffer from the downsides of creating a gazillion types (which probably take more memory and those types stay around until the AppDomain is destroyed).

Finally a tip for Microsoft, move the Cn types out of ThreadLocal, because currently they are also generic (due to the fact that C# automatically makes nested types generic based on the outer type's generic type parameters) and that is unnecessarily wasteful.

Tuesday, 10 November 2009 07:03:31 (W. Europe Standard Time, UTC+01:00)  #    Comments [6]
# Wednesday, 04 November 2009
New Development Snapshot

While the 0.42 release is still baking, a new development snapshot is available.

Changes:

  • Nat worked on clipboard and drag-and-drop support.
  • Volker worked on print support.
  • Various awt fixes/improvements.
  • Code cleanup/refactoring to prepare for IKVM.Reflection (replacement of IKVM.Reflection.Emit).
  • Fixed ikvmstub to ignore private interfaces.
  • Removed .NET 4.0 beta 1 workarounds.
  • Simplified the obj1.getClass() == obj2.getClass() intrinsic to avoid requiring full trust on .NET 4.0.
  • Optimized field reflection. We now delay creating the dynamic methods to access the field until after the field has been accessed a couple of times, this saves a lot of memory for fields that are only usused a few times.
  • Added -publicpackage:<pkg> option to ikvmc. Contributed by Eyal Alaluf of Mainsoft.
  • Added public API to get ClassLoader from Assembly.
  • Added (optional) per-module initialization to custom assembly class loaders.
  • Added ikvmc option -nopeercrossreference and the ability to use -r with peer assemblies.

Binaries available here: ikvmbin-0.43.3595.zip

Wednesday, 04 November 2009 15:23:41 (W. Europe Standard Time, UTC+01:00)  #    Comments [5]
# Tuesday, 27 October 2009
MS09-061 Vulnerability Details

On "Patch Tuesday" two weeks ago Microsoft released security bulletin MS09-061. This bulletin describes three issues, one of which I reported to Microsoft on September 12, 2008. I will describe the details of what is now known as CVE-2009-0091. I have no inside knowledge of the other two vulnerabilities.

As mentioned in the original blog entry, I found the bug while browsing the Rotor sources. Here's the fragment that caught my eye:

    // This method will combine this delegate with the passed delegate
    // to form a new delegate.

    protected override sealed Delegate CombineImpl(Delegate follow)
    {
        // Verify that the types are the same...
        // Actually, we don't need to do this, because Delegate.Combine already checks this.
//      if (!InternalEqualTypes(this, follow)
//          throw new ArgumentException(...)

This is from multicastdelegate.cs (Warning: this link leads to Microsoft Shared Source licensed code).

The code that is commented out is a security check. After seeing this I immediately confirmed (using ildasm) that the, at that time current, production version of mscorlib also didn't include the check. I also checked .NET 1.1 and in that version the check is present. I also checked a pre-release version of Silverlight 2.0 and it also didn't include the check. The subsequent Silverlight 2.0 release on October 14, 2008 included the fix. Microsoft did not find it necessary to credit me with the fix (not even privately).

Why Is This a Security Vulnerability?

In my example type safety exploit, I used a union to bypass type safety. That wasn't an actual security vulnerability because such a union requires full trust. However, if we can combine two different delegate types we can do the same and because of the missing type check this was possible.

If you take TypeSafetyExploitPoC.cs and replace the TypeSystemHole method with the following and add a reference to an assembly containing CombinePoCHelper.il (written in MSIL because that is the easiest way to write your own MulticastDelegate subclass that can call the protected CombineImpl method).

delegate void AnotherDelegate(Union1 u2);

static Union1 TypeSystemHole(Union2 u2)
{
  Union1 u1 = null;
  CombineHelper del1 = delegate { };
  AnotherDelegate del2 = delegate(Union1 u) { u1 = u; };
  del1 = (CombineHelper)CombineHelper.CombineHack(del1, del2);
  del1(u2);
  return u1;
}

Voila!

Tuesday, 27 October 2009 08:17:15 (W. Europe Standard Time, UTC+01:00)  #    Comments [7]
# Monday, 26 October 2009
IKVM 0.42 Release Candidate 2

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.42.0.2.
  • Fix for bug #2883889.
  • Fix for bug #2881954.
  • Fixed automagic serialization interop to work correctly in the face of a __WorkaroundBaseClass__ base type.
  • Small update for .NET 4.0 beta 2.

Binaries available here: ikvmbin-0.42.0.2.zip

Sources: ikvm-0.42.0.2.zip, openjdk6-b16-stripped.zip

Monday, 26 October 2009 06:06:45 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Monday, 19 October 2009
IKVM 0.42 Release Candidate 1

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.42.0.1.
  • Fixed a regression introduced in 0.40 that caused a System.NotSupportedException to be thrown by ikvmc when compiling multiple targets where one target (indirectly) implements a non-public interface from another target. Thanks to Erik Vullings for reporting this.

Binaries available here: ikvmbin-0.42.0.1.zip

Sources: ikvm-0.42.0.1.zip, openjdk6-b16-stripped.zip

Monday, 19 October 2009 06:56:56 (W. Europe Daylight Time, UTC+02:00)  #    Comments [4]
# Monday, 12 October 2009
IKVM 0.42 Release Candidate 0

The first release candidate is available.

Changes since last 0.41 development snapshot:

  • Changed version to 0.42.0.0.
  • Fixed virtual file system regression that caused NullReferenceException when trying to access a non-existing directory.
  • Volker added some print support code.
  • Fixed assertion in ILGenerator.
  • Fixed regression in reflection introduced when we started allowing open generic types to be visible to Java (for stack walking).
  • Fixed bug #2876211.

Binary available here: ikvmbin-0.42.0.0.zip

Sources: ikvm-0.42.0.0.zip, openjdk6-b16-stripped.zip

Monday, 12 October 2009 06:43:34 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 02 October 2009
New Development Snapshot

We're starting to prepare for the 0.42 release. This is the last 0.41 development snapshot.

Changes:

  • Added support for exposing open generic types as Java classes (special "handle" classes that can only be used for stack walking). Fixes bug #2843805.
  • ArrayTypeWrapper: Fixed a race condition and avoid holding the lock while calling external code.
  • Removed vestigial compact framework support code.
  • Some source file restructuring (Moved DynamicTypeWrapper and DotNetTypeWrapper classes into their own source files and AssemblyClassLoader and BootstrapClassLoader clases into AssemblyClassLoader.cs).
  • Fixed regression introduced with recent label handling changes. Bug #2847725.
  • Various AWT fixes by Nat and Volker.
  • Rewrote custom assembly class loader initialization to avoid running user code (static initializer) while holding a lock and to better handle invocation of getClassLoader() during the class loader constructor (or static initializer).
  • Added hack to expose more custom attributes from mscorlib as annotations.

Binary available here: ikvmbin-0.41.3562.zip

Friday, 02 October 2009 06:58:51 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Tuesday, 25 August 2009
New Development Snapshot

Time for another snapshot as there have been a large number of changes since the previous snapshot. The Swing/AWT work that Volker and more recently also Nat have been doing has resulted in SwingSet2 now running quite nicely. Check out these screenshots:

Click for full size Click for full size Click for full size

Note that this is running the OpenJDK Swing code, with the GNU Classpath code the source code view and the HTML tab (the blue "Bouncing Babies!" text is HTML rendered in the tab) never worked. So this is great progress, but a lot more is still needed. Keyboard (and focus) support is still lacking and font support is still fairly limited, for example.

Changes:

  • More AWT/Swing work by Volker Berlin and Nat Luengnaruemitchai.
  • Fixed detection of dynamic assemblies (at runtime).
  • Fixed NullReferenceException when getting annotations on delegate constructor for delegates defined in Java.
  • Added App.config setting (ikvm-emit-symbols) to force emitting debug symbols on or off.
  • Fixed regression introduced in 0.40 that caused ikvmc to crash when specifying a non-public custom class loader.
  • Fixed bug in the handling of Annotation.__ReturnValue and Annotation.__Multiple fake types.
  • Implemented automatic .NET serialization support for Java serializable type (see here for details).
  • Fix for #2829717. Constructing java.lang.String instances from JNI should redirect to static helper method.
  • Several minor IKVM.Reflection.Emit fixes.
  • Added ILGenerator.__GetILOffset().
  • Changed CodeEmitter to use ILGenerator.__GetILOffset() (when usig IKVM.Reflection.Emit) instead of manually tracking the IL offset.
  • Added "clever" exception block assistance mode to ILGenerator. In this mode, leave and endfinally instructions are only auto inserted when necessary.
  • Use ILGenerator's new "clever" mode in ikvmc to produce smaller code.
  • Fixed IKVM.Reflection.Emit to put .pdb file in same location as assembly (instead of current directory). Thanks to Dawid Weiss for reporting this.
  • Removed ISymWrapper.dll dependency from IKVM.Reflection.Emit.PdbWriter.dll.
  • IKVM.Reflection.Emit.PdbWriter.dll now caches the debugging information, instead of "streaming" it into the unmanged PDB writer to workaround the ridiculous memory usage of the unmanaged PDB writer. It is now possible to build the full core class library assemblies set with debugging enabled.
  • Fixed major regression in pdb debugging support introduced in 0.40 that caused local variable support to be completely broken.

Binary available here: ikvmbin-0.41.3524.zip

Tuesday, 25 August 2009 09:21:38 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]