# Monday, 01 December 2008
New Development Snapshot

More ikvmc testing revealed a couple of IKVM.Reflection.Emit bugs and also some bugs that have been lurking in ikvmc for a long time, that only show up under fairly uncommon scenarios when lots of dependencies are missing (and hence the resulting assembly has little chance of working anyway). The fixes are small and relatively low risk, but I'm not sure whether it is worthwhile to back port them to 0.36 and 0.38. Feedback on this is appreciated.

Changes:

  • Added support for SpecialNameAttribute and TypeForwardedToAttribute pseudo custom attributes to IKVM.Reflection.Emit.
  • Fixed IKVM.Reflection.Emit relocation table bug.
  • Fixed IKVM.Reflection.Emit TypeDefOrRef and HasCustomAttribute column size calculations.
  • A couple of ikvmc fixes in dealing with missing classes. This fixes #2351524.
  • Fixed regression introduced in 0.39.3240 that caused failure when reflection triggered a call to a static initializer on a dynamically loaded class.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3257.zip

Monday, 01 December 2008 07:16:54 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Tuesday, 25 November 2008
Pseudo Custom Attributes

One of the really cool things about the CLR, when it was introduced, was the notion of custom attributes. Custom attributes provide an extensible way to add custom metadata. However, since .NET compilers virtually have to support custom attributes, it's tempting to reuse this capability for specifying non-custom metadata. It isn't efficient to encode all metadata in a generic way, hence the pseudo custom attributes were born. From most compilers they appear as normal custom attributes, but the compiler stores them in the metadata in a way that is specific to each pseudo custom attribute.

To recap, my definition of pseudo custom attributes is: Any custom attribute that modifies the metadata, but isn't stored in the CustomAttribute metadata table.

While implementing ikvmc (and later IKVM.Reflection.Emit) I looked for a definitive list of pseudo custom attributes, but there doesn't appear to be one, so I thought I'd make one:

 

ECMA CLI

IKVM.Reflection.Emit

System.Reflection.Emit

AssemblyAlgorithmIdAttribute

Yes*

No

No

AssemblyCultureAttribute

No

No

No

AssemblyFlagsAttribute

Yes*

No

No

AssemblyKeyFileAttribute

No

No

No

AssemblyKeyNameAttribute

No

No

No

AssemblyVersionAttribute

No

No

No

CodeAccessSecurityAttribute**

Yes

No

No

ComImportAttribute

No

Yes

Yes

DefaultParameterValueAttribute

No

Yes

No

DllImportAttribute

Yes

Yes

Yes

FieldOffsetAttribute

Yes

Yes

Yes

InAttribute

Yes

Yes

Yes

MarshalAsAttribute

Yes

Yes

Yes

MethodImplAttribute

Yes

Yes

Yes

NonSerializedAttribute

No

Yes

Yes

OptionalAttribute

No

Yes

Yes

OutAttribute

Yes

Yes

Yes

PreserveSigAttribute

No

Yes

Yes

SerializableAttribute

No***

Yes

Yes

SpecialNameAttribute

No

Yes****

Yes

StructLayoutAttribute

Yes

Yes

Yes

TypeForwardedToAttribute

No

Yes****

No

* Reserved only.
** Its subclasses.
*** Not part of the spec, but the spec does mention it.
**** Not in most recent development snapshot, but code is in cvs.

(System.Reflection.Emit values based on .NET 3.5 SP1)

For completeness and clarity, here is a list of non-pseudo custom attributes that are sometimes mistakenly referred to as pseudo custom attributes:

AllowPartiallyTrustedCallersAttribute

Understood by the runtime, but stored normally.

ClassInterfaceAttribute

Understood by the runtime and compiler, but stored normally.

ComCompatibleVersionAttribute

Understood by the runtime (?) and compiler, but stored normally.

DebuggableAttribute

Understood by the runtime, but stored normally.

GuidAttribute

Understood by the runtime and compiler, but stored normally.

InterfaceTypeAttribute

Understood by the runtime and compiler, but stored normally.

SecurityCriticalAttribute

Understood by the runtime, but stored normally.

SecurityTransparentAttribute

Understood by the runtime, but stored normally.

SecurityTreatAsSafeAttribute

Understood by the runtime, but stored normally.

TypeLibVersionAttribute

Understood by the compiler, but stored normally.

 

Clarifications, corrections and additions to both lists are very much appreciated.

Update: Removed incorrect claim that the runtime only supports the DebuggableAttribute (bool,bool) constructor. Thanks to Kevin M. Owen for the correction.

Tuesday, 25 November 2008 07:04:36 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
# Wednesday, 19 November 2008
Multi Target Support

I've now enabled "multi target" support. This means that you can compile multiple assemblies at once. One of the advantages this has is that it makes dealing with circular dependencies much easier, but even if there aren't any circular dependencies, it can make building much easier because you don't have to do any dependency analysis.

As an example I compiled the jars from the JBoss lib directory:

ikvmc -target:library { commons-codec.jar } { commons-httpclient.jar } { commons-logging.jar } { concurrent.jar } { getopt.jar } { jboss-common.jar } { jboss-jmx.jar } { jboss-system.jar } { jboss-xml-binding.jar } { log4j-boot.jar }

The curly braces define the targets. This is all it takes to compile these jars into seperate assemblies that automatically reference eachother, as required.

Using NDepend (which Patrick Smacchia kindly gave me a free copy of) you can easily graph the dependencies of the resulting assemblies:

In this case there aren't any cycles, but had there been, it would have made no differences.

The multi target feature is by no means done. I want to add a -sharedclassloader option to enable multiple assemblies to share a single Java class loader. I'm also considering some options to select which classes/packages go in which assembly. This would be helpful for splitting the core class library into multiple assemblies.

Changes:

  • Added support to ikvmstub for automatically exporting non-vector array types.
  • Implemented/fixed support for pointers, by ref and non-vector arrays in IKVM.Reflection.Emit.
  • Fixed pointer detection to work for types with multiple indirection levels.
  • Changed proxy stub name mangling to work around ildasm bug.
  • Fixed IKVM.Reflection.Emit to not write token of FAULT pseudo exception that is used as a marker for fault blocks.
  • Fixed class loading to take ClassLoader lock.
  • Changed ikvmc not to generate warning when it loads ikvmstub generated classes.
  • Removed our version of System.Runtime.CompilerServices.ExtensionAttribute and instead add System.Core.jar to the compilation, this will allow us to reference the real ExtensionAttribute when it is available and yet the build will still work (albeit with a warning and without the ExtensionAttribute) when it is not available (i.e. when building on .NET 2.0).
  • Did some clean up and restructuring in IKVM.Reflection.Emit.
  • Fixed sorting of InterfaceImpl table in IKVM.Reflection.Emit.
  • Removed hard coded public keys from JniInterface.cs and Configuration.java.
  • Changed build process so that version number for all assemblies only has to be specified in CommonAssemblyInfo.cs.
  • Enabled multi target support.
  • Fixed typo in IKVM.Reflection.Emit that caused assembly minor version to be set to major version.
  • Added assembly version to ikvmstub.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3245.zip

Wednesday, 19 November 2008 07:46:06 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Monday, 17 November 2008
.NET Array Weirdness

One of the lesser known features of the .NET runtime is the support for multidimensional and/or non-zero-based arrays. The typical arrays that you use (and that are the same as the arrays in Java) are called vectors in the CLI specification, but unfortunately the non-vector arrays don't have a specific name (they are simply called arrays in the CLI specification).

In C# you can easily create a non-vector array:

int[,] arr1 = new int[4, 4]

This creates a two dimensional array of integers with 16 elements. This array is zero based. If you want to create a non-zero-based array, you have to use the API, because C# doesn't directly support that:

int[,] arr2 = (int[,])Array.CreateInstance(typeof(int), new[] { 4, 4 }, new[] { 1, 1 });

This also creates a two dimensional array of integers with 16 elements, but in this case the indexes run from 1 through 5.

So far, so good. Now let's look at how things look at the IL level. If you use ildasm to look at the first C# based instantation you'll see:

ldc.i4.1
ldc.i4.1
newobj instance void int32[0...,0...]::.ctor(int32, int32)

Most of this looks straightforward if you're familiar with IL, except for the [0...,0...] part. It looks like the lower bounds are part of the type, but a simple experiment shows that this is not the case:

Console.WriteLine(arr1.GetType() == arr2.GetType());

This prints out True. This implies that the lower bounds are not part of the type (and the CLI specification confirms this). So why are they part of the signature? I don't know for sure, but it does have an interesting consequence. You can overload methods based on this:

.assembly extern mscorlib { }
.assembly Test { }
.module Test.exe

.class public Program
{
  .method public static void Foo(int32[0...,0...] foo)
  {
    ret
  }

  .method public static void Foo(int32[1...,1...] foo)
  {
    ret
  }

  .method private static void Main()
  {
    .entrypoint
    ldc.i4.1
    ldc.i4.1
    newobj instance void int32[0...,0...]::.ctor(int32, int32)
    call void Program::Foo(int32[0...,0...])
    ret
  }
}

This is a valid and verifiable application. Unfortunately, using the  .NET reflection API it is impossible to see the difference in signatures between the two Foo methods.

This limitation in reflection means that code compiled with the new IKVM.Reflection.Emit based ikvmc won't be able to call methods that have non-zero-based array parameters. It is also impossible to override these methods, but that was already the case with the previous System.Reflection.Emit based implementation as well.

Finally, in the above text I talk about the lower bounds, but the same thing applies to the upper bounds of the array.

Monday, 17 November 2008 07:08:46 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Friday, 14 November 2008
Introducing IKVM.Reflection.Emit

Over the past two months I've been working on reimplementing a large portion of the Reflection.Emit API from scratch. After finally growing tired of the System.Reflection.Emit bugs and limitations and not finding Mono.Cecil satisfactory either, I decided to build my own implementation.

I started out with these design goals:

  • API compatible with System.Reflection.Emit (as much as possible).
  • Implement only the Emit part and be compatible with System.Reflection.
  • Only implement functionality required by ikvmc, but not implemented functionality shouldn't silently fail (i.e. it should throw a NotSupportedException or NotImplementedException). This also means that the API is mostly write-only.
  • Efficient implementation, optimized for ikvmc specific scenarios.

I think I've met or exceeded all of the design goals. Without doing any significant performance work on my Ref.Emit implementation (other than the design), ikvmc became so much faster that it is rather emberassing for the Microsoft System.Reflection.Emit implementation.

I've only had to make a couple of minor changes to the ikvmc sources (apart from changing using System.Reflection.Emit; to using IKVM.Reflection.Emit; in every file) to account for the fact that IKVM.Reflection.Emit.ModuleBuilder and IKVM.Reflection.Emit.AssemblyBuilder unfortunately cannot extend System.Reflection.Module and System.Reflection.Assembly. However, it looks like this is fixed in the .NET 4.0 CTP.

Here are some random statistics about compiling IKVM.OpenJDK.ClassLibrary.dll on .NET 2.0 SP2 x64:

  System.Reflection.Emit     IKVM.Reflection.Emit  
File size 31,645,696 30,480,896   bytes
CPU time used 272 35   seconds
Peak virtual memory 1,433,399,296 1,035,018,240   bytes
Generation 0 GCs 770 896  
Generation 1 GCs 201 240  
Generation 2 GCs 11 8  

(The huge memory usage is not because it actually needs that much memory, but simply the result of the fact that garbage collection is more efficient if you have more memory available and that my system had about 1.5GB of free memory while running these tests.)

The smaller file size is because System.Reflection.Emit always uses fat method headers and IKVM.Reflection.Emit uses tiny method headers whenever possible.

There is still some work left to do, I've only spent limited time on debugging support and there is no support for Mono's .mdb format yet. I also haven't done any testing on Mono yet.

BTW, thanks to Sebastien Pouliot for code I lifted from Mono to parse strong name CAPI key blobs.

Other changes in this snapshot:

  • Dropped support for Visual Studio 2005.
  • Added error message when map.xml references non-existing constructor.
  • Added more statistics to ikvmc -time option output.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3240.zip

Friday, 14 November 2008 10:01:50 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
# Wednesday, 12 November 2008
IKVM 0.38 Release Candidate 2

A codegen bug was found (not a regression, so there will be a 0.36 update as well) so here's another release candidate.

Changes:

  • Changed version to 0.38.0.2.
  • Fixed openjdk.build BOM issue on Linux.
  • Fixed jsr verifier bug that caused incorrect codegen under very specific circumstances (thanks to Brian Schwallier for tracking down a repro).

Binaries available here: ikvmbin-0.38.0.2.zip
Sources (+ binaries): Sources: ikvm-0.38.0.2.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip

Wednesday, 12 November 2008 06:13:29 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Wednesday, 05 November 2008
IKVM 0.36 Update 2 Release Candidate 2

As announced when 0.36 was released, I will periodically release updates to 0.36 as long as there is enough interest in .NET 1.1 support.

This is the second release candidate of the second update.

Changes (all are back ported fixes):

  • Changed version to 0.36.0.13.
  • Fixed ikvmc not to open the key file for write access.
  • Added more efficient float/double to/from int/long bits converters.
  • Fixed libikvm-native.so build to include reference to gmodule-2.0 library.
  • Fixed ikvmc not to open the key file for write access.
  • Fixed Graphics2D.rotate() to convert rotation angle from radians (Java) to degrees (.NET).
  • Applied awt patch #1979656 by Daniel Wilson.
  • Fixed three String bugs found by OpenJDK string tests.
  • Fixed ldc <class> where <class> is a ghost array.
  • Fixed bug in instanceof <class> where <class> is a Serializable[].
  • Removed incorrect DataFormatException thrown in java.util.zip.InflaterHuffmanTree.
  • Fixed #2001802 contributed by Andy Malakov.
  • Fixed #2001799.
  • Fixed #2006953.
  • Made finalize() and clone() methods in cli.System.Object and cli.System.Exception final.

Binaries available here: ikvmbin-0.36.0.13.zip
Sources (+ binaries): ikvm-0.36.0.13.zip

Wednesday, 05 November 2008 10:08:46 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Tuesday, 04 November 2008
IKVM 0.38 Release Candidate 1

A couple of minor fixes.

Changes since RC0:

  • Updated version to 0.38.0.1
  • Hide nested types generated by callerID support
  • Made finalize() and clone() methods in cli.System.Object and cli.System.Exception final
  • Fixed reflection to work on .NET 2.0 RTM

Note that even though I fixed reflection to work on .NET 2.0 RTM, it still isn't a supported platform, I strongly recommend .NET 2.0 SP1 or higher.

Binaries available here: ikvmbin-0.38.0.1.zip

Sources: ikvm-0.38.0.1.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip

Tuesday, 04 November 2008 06:36:51 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Friday, 26 September 2008
More Reflection.Emit Brokenness

Pardon me if I sound a little bitter today, but I just wasted almost a full day trying to work around this bug only to be stopped by yet another bug that makes it impossible to generate two mutually dependent assemblies with Reflection.Emit.

Also, while debugging I noticed another mind bogglingly stupid bug in the System.Reflection.Assembly source:

public override int GetHashCode() { return base.GetHashCode(); }

Why would you want to do that? Oh, of course! It's to get rid of a compiler warning... If you override Equals() you should also override GetHashCode(), BUT NOT LIKE THIS.

Here's small program that demonstrates the problem:

using System;
using System.Reflection;
using System.Reflection.Emit;

class Program
{
  static void Main()
  {
    AssemblyBuilder ab1 = AppDomain.CurrentDomain.DefineDynamicAssembly(new AssemblyName("A"),
                            AssemblyBuilderAccess.Run);
    ModuleBuilder mod1 = ab1.DefineDynamicModule("A.dll");
    TypeBuilder tb1 = mod1.DefineType("T");
    Type type = tb1.CreateType();
    Console.WriteLine(ab1.Equals(type.Assembly));
    Console.WriteLine(ab1.GetHashCode() == type.Assembly.GetHashCode());
  }
}

This prints out:

True
False

That clearly violates the Object.GetHashCode() contract.

Friday, 26 September 2008 08:47:51 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]
# Wednesday, 24 September 2008
IKVM 0.38 Release Candidate 0

As with previous release candidates, this release includes strong named binaries and is considered to be (nearly) ready for production use. Please test this version and give feedback as soon as possible.

Changes since previous snapshot:

  • Changed version to 0.38.0.0 and strong named binaries.
  • Added missing HTMLEntities.res resource.
  • Re-introduced workaround for .NET JIT bug that causes .cctor not to run when a DynamicMethod invokes a method or gets/sets a field.

Binaries available here: ikvmbin-0.38.0.0.zip

Sources: ikvm-0.38.0.0.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip

Wednesday, 24 September 2008 08:07:13 (W. Europe Daylight Time, UTC+02:00)  #    Comments [2]