# Tuesday, 10 November 2009
Going Crazy with Generics, or The Story of ThreadLocal(Of T)

Jon Skeet recently blogged about the performance of [ThreadStatic] versus the new .NET 4.0 ThreadLocal<T>. I was surprised to see that ThreadLocal<T> was faster than [ThreadStatic], because ThreadLocal<T> uses [ThreadStatic] as the underlying primitive.

How do you go from a static field to a per instance field? It's simple, once you think of it. You (ab)use generic types. Here's a simplified ThreadLocal<T>:

public class ThreadLocal<T>
{
  HolderBase holder;
  static int count;
  static Type[] types = new Type[] { typeof(C1), typeof(C2), typeof(C3) };

  abstract class HolderBase
  {
    internal abstract T Value { get; set; }
  }

  class C1 { }
  class C2 { }
  class C3 { }

  class Holder : HolderBase
  {
    [ThreadStatic]
    static T val;

    internal override T Value
    {
      get { return val; }
      set { val = value; }
    }
  }

  public ThreadLocal()
  {
    holder = MakeHolder(Interlocked.Increment(ref count) - 1);
  }

  HolderBase MakeHolder(int index)
  {
    Type t1 = types[index % 3];
    Type t2 = types[(index / 3) % 3];
    Type t3 = types[index / 9];
    Type type = typeof(Holder<,,>).MakeGenericType(typeof(T), t1, t2, t3);
    return (HolderBase)Activator.CreateInstance(type);
  }

  public T Value
  {
    get { return holder.Value; }
    set { holder.Value = value; }
  }
}

The real ThreadLocal<T> type in .NET 4.0 beta 2 is much more complex, because it has to deal with recycling the types and protecting against returning a value from a recycled type. It also uses a higher base counting system  to number the types, the maximum number of types generated (per T) is 4096 in beta 2. After you allocate more than that, it falls back to using a holder type that uses Thread.SetData().

I'm not sure what to make of this. It's a clever trick, but I think it ultimately is too clever. I benchmarked a simpler approach using arrays (where each ThreadLocal<T> simply allocated an index in the [ThreadStatic] array) and it was a little bit faster and doesn't suffer from the downsides of creating a gazillion types (which probably take more memory and those types stay around until the AppDomain is destroyed).

Finally a tip for Microsoft, move the Cn types out of ThreadLocal, because currently they are also generic (due to the fact that C# automatically makes nested types generic based on the outer type's generic type parameters) and that is unnecessarily wasteful.

Tuesday, 10 November 2009 07:03:31 (W. Europe Standard Time, UTC+01:00)  #    Comments [6]
# Wednesday, 04 November 2009
New Development Snapshot

While the 0.42 release is still baking, a new development snapshot is available.

Changes:

  • Nat worked on clipboard and drag-and-drop support.
  • Volker worked on print support.
  • Various awt fixes/improvements.
  • Code cleanup/refactoring to prepare for IKVM.Reflection (replacement of IKVM.Reflection.Emit).
  • Fixed ikvmstub to ignore private interfaces.
  • Removed .NET 4.0 beta 1 workarounds.
  • Simplified the obj1.getClass() == obj2.getClass() intrinsic to avoid requiring full trust on .NET 4.0.
  • Optimized field reflection. We now delay creating the dynamic methods to access the field until after the field has been accessed a couple of times, this saves a lot of memory for fields that are only usused a few times.
  • Added -publicpackage:<pkg> option to ikvmc. Contributed by Eyal Alaluf of Mainsoft.
  • Added public API to get ClassLoader from Assembly.
  • Added (optional) per-module initialization to custom assembly class loaders.
  • Added ikvmc option -nopeercrossreference and the ability to use -r with peer assemblies.

Binaries available here: ikvmbin-0.43.3595.zip

Wednesday, 04 November 2009 15:23:41 (W. Europe Standard Time, UTC+01:00)  #    Comments [5]
# Tuesday, 27 October 2009
MS09-061 Vulnerability Details

On "Patch Tuesday" two weeks ago Microsoft released security bulletin MS09-061. This bulletin describes three issues, one of which I reported to Microsoft on September 12, 2008. I will describe the details of what is now known as CVE-2009-0091. I have no inside knowledge of the other two vulnerabilities.

As mentioned in the original blog entry, I found the bug while browsing the Rotor sources. Here's the fragment that caught my eye:

    // This method will combine this delegate with the passed delegate
    // to form a new delegate.

    protected override sealed Delegate CombineImpl(Delegate follow)
    {
        // Verify that the types are the same...
        // Actually, we don't need to do this, because Delegate.Combine already checks this.
//      if (!InternalEqualTypes(this, follow)
//          throw new ArgumentException(...)

This is from multicastdelegate.cs (Warning: this link leads to Microsoft Shared Source licensed code).

The code that is commented out is a security check. After seeing this I immediately confirmed (using ildasm) that the, at that time current, production version of mscorlib also didn't include the check. I also checked .NET 1.1 and in that version the check is present. I also checked a pre-release version of Silverlight 2.0 and it also didn't include the check. The subsequent Silverlight 2.0 release on October 14, 2008 included the fix. Microsoft did not find it necessary to credit me with the fix (not even privately).

Why Is This a Security Vulnerability?

In my example type safety exploit, I used a union to bypass type safety. That wasn't an actual security vulnerability because such a union requires full trust. However, if we can combine two different delegate types we can do the same and because of the missing type check this was possible.

If you take TypeSafetyExploitPoC.cs and replace the TypeSystemHole method with the following and add a reference to an assembly containing CombinePoCHelper.il (written in MSIL because that is the easiest way to write your own MulticastDelegate subclass that can call the protected CombineImpl method).

delegate void AnotherDelegate(Union1 u2);

static Union1 TypeSystemHole(Union2 u2)
{
  Union1 u1 = null;
  CombineHelper del1 = delegate { };
  AnotherDelegate del2 = delegate(Union1 u) { u1 = u; };
  del1 = (CombineHelper)CombineHelper.CombineHack(del1, del2);
  del1(u2);
  return u1;
}

Voila!

Tuesday, 27 October 2009 08:17:15 (W. Europe Standard Time, UTC+01:00)  #    Comments [7]
# Monday, 26 October 2009
IKVM 0.42 Release Candidate 2

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.42.0.2.
  • Fix for bug #2883889.
  • Fix for bug #2881954.
  • Fixed automagic serialization interop to work correctly in the face of a __WorkaroundBaseClass__ base type.
  • Small update for .NET 4.0 beta 2.

Binaries available here: ikvmbin-0.42.0.2.zip

Sources: ikvm-0.42.0.2.zip, openjdk6-b16-stripped.zip

Monday, 26 October 2009 06:06:45 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Monday, 19 October 2009
IKVM 0.42 Release Candidate 1

A new release candidate is available.

Changes since previous release candidate:

  • Changed version to 0.42.0.1.
  • Fixed a regression introduced in 0.40 that caused a System.NotSupportedException to be thrown by ikvmc when compiling multiple targets where one target (indirectly) implements a non-public interface from another target. Thanks to Erik Vullings for reporting this.

Binaries available here: ikvmbin-0.42.0.1.zip

Sources: ikvm-0.42.0.1.zip, openjdk6-b16-stripped.zip

Monday, 19 October 2009 06:56:56 (W. Europe Daylight Time, UTC+02:00)  #    Comments [4]
# Monday, 12 October 2009
IKVM 0.42 Release Candidate 0

The first release candidate is available.

Changes since last 0.41 development snapshot:

  • Changed version to 0.42.0.0.
  • Fixed virtual file system regression that caused NullReferenceException when trying to access a non-existing directory.
  • Volker added some print support code.
  • Fixed assertion in ILGenerator.
  • Fixed regression in reflection introduced when we started allowing open generic types to be visible to Java (for stack walking).
  • Fixed bug #2876211.

Binary available here: ikvmbin-0.42.0.0.zip

Sources: ikvm-0.42.0.0.zip, openjdk6-b16-stripped.zip

Monday, 12 October 2009 06:43:34 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Friday, 02 October 2009
New Development Snapshot

We're starting to prepare for the 0.42 release. This is the last 0.41 development snapshot.

Changes:

  • Added support for exposing open generic types as Java classes (special "handle" classes that can only be used for stack walking). Fixes bug #2843805.
  • ArrayTypeWrapper: Fixed a race condition and avoid holding the lock while calling external code.
  • Removed vestigial compact framework support code.
  • Some source file restructuring (Moved DynamicTypeWrapper and DotNetTypeWrapper classes into their own source files and AssemblyClassLoader and BootstrapClassLoader clases into AssemblyClassLoader.cs).
  • Fixed regression introduced with recent label handling changes. Bug #2847725.
  • Various AWT fixes by Nat and Volker.
  • Rewrote custom assembly class loader initialization to avoid running user code (static initializer) while holding a lock and to better handle invocation of getClassLoader() during the class loader constructor (or static initializer).
  • Added hack to expose more custom attributes from mscorlib as annotations.

Binary available here: ikvmbin-0.41.3562.zip

Friday, 02 October 2009 06:58:51 (W. Europe Daylight Time, UTC+02:00)  #    Comments [1]
# Tuesday, 25 August 2009
New Development Snapshot

Time for another snapshot as there have been a large number of changes since the previous snapshot. The Swing/AWT work that Volker and more recently also Nat have been doing has resulted in SwingSet2 now running quite nicely. Check out these screenshots:

Click for full size Click for full size Click for full size

Note that this is running the OpenJDK Swing code, with the GNU Classpath code the source code view and the HTML tab (the blue "Bouncing Babies!" text is HTML rendered in the tab) never worked. So this is great progress, but a lot more is still needed. Keyboard (and focus) support is still lacking and font support is still fairly limited, for example.

Changes:

  • More AWT/Swing work by Volker Berlin and Nat Luengnaruemitchai.
  • Fixed detection of dynamic assemblies (at runtime).
  • Fixed NullReferenceException when getting annotations on delegate constructor for delegates defined in Java.
  • Added App.config setting (ikvm-emit-symbols) to force emitting debug symbols on or off.
  • Fixed regression introduced in 0.40 that caused ikvmc to crash when specifying a non-public custom class loader.
  • Fixed bug in the handling of Annotation.__ReturnValue and Annotation.__Multiple fake types.
  • Implemented automatic .NET serialization support for Java serializable type (see here for details).
  • Fix for #2829717. Constructing java.lang.String instances from JNI should redirect to static helper method.
  • Several minor IKVM.Reflection.Emit fixes.
  • Added ILGenerator.__GetILOffset().
  • Changed CodeEmitter to use ILGenerator.__GetILOffset() (when usig IKVM.Reflection.Emit) instead of manually tracking the IL offset.
  • Added "clever" exception block assistance mode to ILGenerator. In this mode, leave and endfinally instructions are only auto inserted when necessary.
  • Use ILGenerator's new "clever" mode in ikvmc to produce smaller code.
  • Fixed IKVM.Reflection.Emit to put .pdb file in same location as assembly (instead of current directory). Thanks to Dawid Weiss for reporting this.
  • Removed ISymWrapper.dll dependency from IKVM.Reflection.Emit.PdbWriter.dll.
  • IKVM.Reflection.Emit.PdbWriter.dll now caches the debugging information, instead of "streaming" it into the unmanged PDB writer to workaround the ridiculous memory usage of the unmanaged PDB writer. It is now possible to build the full core class library assemblies set with debugging enabled.
  • Fixed major regression in pdb debugging support introduced in 0.40 that caused local variable support to be completely broken.

Binary available here: ikvmbin-0.41.3524.zip

Tuesday, 25 August 2009 09:21:38 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Monday, 24 August 2009
Microsoft PDC

If you're going to the PDC and want to chat, let me know.

Meet me in Los Angeles -- PDC 2009

Monday, 24 August 2009 06:23:45 (W. Europe Daylight Time, UTC+02:00)  #    Comments [0]
# Tuesday, 11 August 2009
Serialization Interop

I'm not a big fan of serialization, but there is one use that makes sense to me; intra-process, cross-AppDomain serialization. If you have ever done any cross-AppDomain work with IKVM you've probably run into the situation where a Java exception couldn't be serialized across the AppDomain boundary.

I've finally addressed this by building automatic (oneway) serialization interop support into IKVM. This means that most Java classes that are serializable should now automatically become .NET serializable. There are, however, some important caveats:

  • Serialized streams will not be compatible between different IKVM releases. This is intended *only* for cross-AppDomain serialization between different instances of the same IKVM code.
  • ObjectOutputStream.writeUnshared() and ObjectInputStream.readUnshared() have not been implemented. So any classes that rely on those will fail to serialize/deserialize.
  • Instances of dynamically loaded Java classes and statically (ikvmc) compiled classes that implement readResolve() will fail deserialization if they (directly or indirectly) serialize self references.
  • Deserialization ordering may be different, meaning that if a class has a custom deserialization method, it may encounter objects that have not been completely deserialized.
  • Under some circumstances, a class that implements readResolve() may have its readResolve() method called twice on the same object.
  • When a ghost array is serialized, it is serialized as an object[] (i.e. it loses its specific type).

Again, any Java class that is serializable (i.e. that implements java.io.Serializable or java.io.Externalizable and follows the associated rules) is generally automatically .NET serializable. There are a couple of exceptions where ikvmc and the runtime will assume that your class wants to handle its own .NET serialization:

Using the .NET custom serialization custom attributes OnDeserializedAttribute, OnDeserializingAttribute, OnSerializedAttribute or OnSerializingAttribute does not interfere with getting automatic serialization support (and the .NET serialization engine will call the annotated methods at the appropriate times).

Inheritance - Extending Java classes in .NET

When you want to subclass a serializable Java class in (e.g.) C# and make your subclass serializable as well, you need to do two simple things:

  1. Add a [Serializable] attribute to your class.
  2. Add a .NET deserialization constructor that calls the base class constructor and does nothing else.

Here's an example of extending java.util.ArrayList:

[Serializable]
class MyList : java.util.ArrayList
{
  private int exampleField;

  public MyList()
  {
  }

  protected MyList(SerializationInfo info, StreamingContext context)
    : base(info, context)
  {
  }
}

MyList is now .NET serializable and exampleField will automatically be serialized/deserialized.

Inheritance - Extending .NET classes in Java

For this scenario, nothing has really changed. You still need to follow the standard .NET rules for creating serializable types.

Serializing .NET objects in Java

The automatic serialization interop is only one way, so you won't be able to serialize .NET objects using Java serialization (unless they happen to implement java.io.Serializable.__Interface or java.io.Externalizable). I currently have no plans to implement this functionality.

Tuesday, 11 August 2009 09:24:07 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]