# Friday, 19 December 2008
Modularization Status Update

I've been doing lots of modularization work, but I'm not quite ready to check it in yet. I've now split the class library assembly into ten assemblies and I've got "shared class loader" support working (at least for the core class library scenario).

I've been positively surprised by how many scenarios can be supported while only loading IKVM.OpenJDK.Core.dll. As the graph below shows, it has more (necessarily circular) dependencies than I would like, but for a number of scenarios I've carefully tweaked the set of classes in Core to enable lazy loading the dependencies only when they are really needed.

I'm not yet ready to commit to supported "Core-only" scenarios, but here's a flavor of some things I've been able to do:

  • Running "Hello, World!" in dynamic mode
  • Serialization
  • Reflection
  • File I/O & Socket I/O (both classic and nio)

Packages included in Core:

  • java.io
  • java.lang
  • java.lang.annotation
  • java.lang.ref
  • java.lang.reflection
  • java.math
  • java.net
  • java.nio
  • java.nio.channels
  • java.nio.channels.spi
  • java.nio.charset
  • java.nio.charset.spi
  • java.security
  • java.security.cert
  • java.util
  • java.util.concurrent
  • java.util.concurrent.atomic
  • java.util.concurrent.locks
  • java.util.regex
  • javax.net

Here is the assembly dependency graph (click the image to enlarge):

Here are the current assembly file sizes:

IKVM.OpenJDK.Core.dll 3,278,848
IKVM.OpenJDK.Security.dll 2,646,016
IKVM.OpenJDK.Util.dll 1,111,040
IKVM.OpenJDK.Xml.dll 8,497,664
IKVM.OpenJDK.SwingAWT.dll 3,040,768
IKVM.OpenJDK.Charsets.dll 5,017,088
IKVM.OpenJDK.Corba.dll 2,335,232
IKVM.OpenJDK.Management.dll      1,180,160
IKVM.OpenJDK.Misc.dll 2,668,032
IKVM.OpenJDK.Text.dll 628,736
  30,403,584

It's interesting to see that the total is slightly less than the previous size of IKVM.OpenJDK.ClassLibrary.dll (30,472,704).

Friday, 19 December 2008 08:49:00 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Monday, 08 December 2008
Modularizing OpenJDK

Recently Mark Reinhold started blogging about modularizing the JDK. I've been getting requests to split up IKVM.OpenJDK.ClassLibrary.dll for a long time. It's good to see that the Java platform is also moving in that direction, but we needn't wait for that.

I've been working on making ikvmc support "multi target" mode for a while now. Last week I used this mode to compile the OpenJDK classes into 743 different assemblies (one per package) . I then tried to use NDepend to dig through the dependencies, but it turns out that 743 assemblies is a bit too much to handle (NDepend handled it resonably well, but the dependency graph was way too large to be useful). So I started moving some obvious stuff together. The result was this NDepend report. Still a lot of data and a lot of dependencies, but some patterns are starting to emerge.

Here's my preliminary view of how things could be split:

IKVM.OpenJDK.Core.dll 5 MB
IKVM.OpenJDK.Security.dll 3 MB
IKVM.OpenJDK.Util.dll 1 MB
IKVM.OpenJDK.Xml.dll 8 MB
IKVM.OpenJDK.SwingAWT.dll 3 MB
IKVM.OpenJDK.Charsets.dll 5 MB
IKVM.OpenJDK.Corba.dll 2 MB
IKVM.OpenJDK.Management.dll      2 MB
IKVM.OpenJDK.Misc.dll 3 MB

(The sizes are only approximate.)

I had originally hoped to make IKVM.OpenJDK.Core.dll smaller by keeping java.util, java.net and java.io out of it, but it looks like you won't be able to run anything non-trivial without requiring classes from these packages, so it makes more sense to put them into the core assembly. The IKVM.OpenJDK.Security.dll and  IKVM.OpenJDK.Util.dll assemblies contain other util and security related packages that shouldn't be needed as often.

It is possible to split packages across assemblies (e.g. java.awt.AWTPermission will be in IKVM.OpenJDK.Core.dll because java.lang.SecurityManager depends on it), but given the potential for confusion my current thinking is that it is probably best to only move individual classes into Core should it be necessary (because, realistically, you can't develop without having a reference to Core you're less likely to be confused when trying to locate the class).

To avoid confusion or expectations that are too high: I haven't yet built the runtime infrastructure to support this. So while I can compile the class library into all these parts, the runtime won't actually be able to work correctly, because it still expects all the boot class loader classes in a single assembly.

As always, feedback on the proposed split is very welcome.

NDepend

I really like NDepend's ability to show the dependencies in different ways and the interactive dependency matrix that allows you to drill down into a dependency to see exactly where it comes from. Beyond dependency analysis it also has a power SQL like query language to allows use to query dependencies and compute all the code metrics you can come up with. It also includes a number of code metrics out of the box, but those aren't really my cup of tea, so I can't comment how useful they are.

One other small but really nice thing about it is that you can run it without installing. IMO this is very nice compared with installers that do who knows what to your system (and require administrator access).

An evaluation copy can be downloaded from their website.

Full disclosure: I was given a free copy of NDepend Professional Edition.

Monday, 08 December 2008 06:51:28 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Monday, 01 December 2008
New Development Snapshot

More ikvmc testing revealed a couple of IKVM.Reflection.Emit bugs and also some bugs that have been lurking in ikvmc for a long time, that only show up under fairly uncommon scenarios when lots of dependencies are missing (and hence the resulting assembly has little chance of working anyway). The fixes are small and relatively low risk, but I'm not sure whether it is worthwhile to back port them to 0.36 and 0.38. Feedback on this is appreciated.

Changes:

  • Added support for SpecialNameAttribute and TypeForwardedToAttribute pseudo custom attributes to IKVM.Reflection.Emit.
  • Fixed IKVM.Reflection.Emit relocation table bug.
  • Fixed IKVM.Reflection.Emit TypeDefOrRef and HasCustomAttribute column size calculations.
  • A couple of ikvmc fixes in dealing with missing classes. This fixes #2351524.
  • Fixed regression introduced in 0.39.3240 that caused failure when reflection triggered a call to a static initializer on a dynamically loaded class.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3257.zip

Monday, 01 December 2008 07:16:54 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Tuesday, 25 November 2008
Pseudo Custom Attributes

One of the really cool things about the CLR, when it was introduced, was the notion of custom attributes. Custom attributes provide an extensible way to add custom metadata. However, since .NET compilers virtually have to support custom attributes, it's tempting to reuse this capability for specifying non-custom metadata. It isn't efficient to encode all metadata in a generic way, hence the pseudo custom attributes were born. From most compilers they appear as normal custom attributes, but the compiler stores them in the metadata in a way that is specific to each pseudo custom attribute.

To recap, my definition of pseudo custom attributes is: Any custom attribute that modifies the metadata, but isn't stored in the CustomAttribute metadata table.

While implementing ikvmc (and later IKVM.Reflection.Emit) I looked for a definitive list of pseudo custom attributes, but there doesn't appear to be one, so I thought I'd make one:

 

ECMA CLI

IKVM.Reflection.Emit

System.Reflection.Emit

AssemblyAlgorithmIdAttribute

Yes*

No

No

AssemblyCultureAttribute

No

No

No

AssemblyFlagsAttribute

Yes*

No

No

AssemblyKeyFileAttribute

No

No

No

AssemblyKeyNameAttribute

No

No

No

AssemblyVersionAttribute

No

No

No

CodeAccessSecurityAttribute**

Yes

No

No

ComImportAttribute

No

Yes

Yes

DefaultParameterValueAttribute

No

Yes

No

DllImportAttribute

Yes

Yes

Yes

FieldOffsetAttribute

Yes

Yes

Yes

InAttribute

Yes

Yes

Yes

MarshalAsAttribute

Yes

Yes

Yes

MethodImplAttribute

Yes

Yes

Yes

NonSerializedAttribute

No

Yes

Yes

OptionalAttribute

No

Yes

Yes

OutAttribute

Yes

Yes

Yes

PreserveSigAttribute

No

Yes

Yes

SerializableAttribute

No***

Yes

Yes

SpecialNameAttribute

No

Yes****

Yes

StructLayoutAttribute

Yes

Yes

Yes

TypeForwardedToAttribute

No

Yes****

No

* Reserved only.
** Its subclasses.
*** Not part of the spec, but the spec does mention it.
**** Not in most recent development snapshot, but code is in cvs.

(System.Reflection.Emit values based on .NET 3.5 SP1)

For completeness and clarity, here is a list of non-pseudo custom attributes that are sometimes mistakenly referred to as pseudo custom attributes:

AllowPartiallyTrustedCallersAttribute

Understood by the runtime, but stored normally.

ClassInterfaceAttribute

Understood by the runtime and compiler, but stored normally.

ComCompatibleVersionAttribute

Understood by the runtime (?) and compiler, but stored normally.

DebuggableAttribute

Understood by the runtime, but stored normally.

GuidAttribute

Understood by the runtime and compiler, but stored normally.

InterfaceTypeAttribute

Understood by the runtime and compiler, but stored normally.

SecurityCriticalAttribute

Understood by the runtime, but stored normally.

SecurityTransparentAttribute

Understood by the runtime, but stored normally.

SecurityTreatAsSafeAttribute

Understood by the runtime, but stored normally.

TypeLibVersionAttribute

Understood by the compiler, but stored normally.

 

Clarifications, corrections and additions to both lists are very much appreciated.

Update: Removed incorrect claim that the runtime only supports the DebuggableAttribute (bool,bool) constructor. Thanks to Kevin M. Owen for the correction.

Tuesday, 25 November 2008 07:04:36 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
# Wednesday, 19 November 2008
Multi Target Support

I've now enabled "multi target" support. This means that you can compile multiple assemblies at once. One of the advantages this has is that it makes dealing with circular dependencies much easier, but even if there aren't any circular dependencies, it can make building much easier because you don't have to do any dependency analysis.

As an example I compiled the jars from the JBoss lib directory:

ikvmc -target:library { commons-codec.jar } { commons-httpclient.jar } { commons-logging.jar } { concurrent.jar } { getopt.jar } { jboss-common.jar } { jboss-jmx.jar } { jboss-system.jar } { jboss-xml-binding.jar } { log4j-boot.jar }

The curly braces define the targets. This is all it takes to compile these jars into seperate assemblies that automatically reference eachother, as required.

Using NDepend (which Patrick Smacchia kindly gave me a free copy of) you can easily graph the dependencies of the resulting assemblies:

In this case there aren't any cycles, but had there been, it would have made no differences.

The multi target feature is by no means done. I want to add a -sharedclassloader option to enable multiple assemblies to share a single Java class loader. I'm also considering some options to select which classes/packages go in which assembly. This would be helpful for splitting the core class library into multiple assemblies.

Changes:

  • Added support to ikvmstub for automatically exporting non-vector array types.
  • Implemented/fixed support for pointers, by ref and non-vector arrays in IKVM.Reflection.Emit.
  • Fixed pointer detection to work for types with multiple indirection levels.
  • Changed proxy stub name mangling to work around ildasm bug.
  • Fixed IKVM.Reflection.Emit to not write token of FAULT pseudo exception that is used as a marker for fault blocks.
  • Fixed class loading to take ClassLoader lock.
  • Changed ikvmc not to generate warning when it loads ikvmstub generated classes.
  • Removed our version of System.Runtime.CompilerServices.ExtensionAttribute and instead add System.Core.jar to the compilation, this will allow us to reference the real ExtensionAttribute when it is available and yet the build will still work (albeit with a warning and without the ExtensionAttribute) when it is not available (i.e. when building on .NET 2.0).
  • Did some clean up and restructuring in IKVM.Reflection.Emit.
  • Fixed sorting of InterfaceImpl table in IKVM.Reflection.Emit.
  • Removed hard coded public keys from JniInterface.cs and Configuration.java.
  • Changed build process so that version number for all assemblies only has to be specified in CommonAssemblyInfo.cs.
  • Enabled multi target support.
  • Fixed typo in IKVM.Reflection.Emit that caused assembly minor version to be set to major version.
  • Added assembly version to ikvmstub.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3245.zip

Wednesday, 19 November 2008 07:46:06 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Monday, 17 November 2008
.NET Array Weirdness

One of the lesser known features of the .NET runtime is the support for multidimensional and/or non-zero-based arrays. The typical arrays that you use (and that are the same as the arrays in Java) are called vectors in the CLI specification, but unfortunately the non-vector arrays don't have a specific name (they are simply called arrays in the CLI specification).

In C# you can easily create a non-vector array:

int[,] arr1 = new int[4, 4]

This creates a two dimensional array of integers with 16 elements. This array is zero based. If you want to create a non-zero-based array, you have to use the API, because C# doesn't directly support that:

int[,] arr2 = (int[,])Array.CreateInstance(typeof(int), new[] { 4, 4 }, new[] { 1, 1 });

This also creates a two dimensional array of integers with 16 elements, but in this case the indexes run from 1 through 5.

So far, so good. Now let's look at how things look at the IL level. If you use ildasm to look at the first C# based instantation you'll see:

ldc.i4.1
ldc.i4.1
newobj instance void int32[0...,0...]::.ctor(int32, int32)

Most of this looks straightforward if you're familiar with IL, except for the [0...,0...] part. It looks like the lower bounds are part of the type, but a simple experiment shows that this is not the case:

Console.WriteLine(arr1.GetType() == arr2.GetType());

This prints out True. This implies that the lower bounds are not part of the type (and the CLI specification confirms this). So why are they part of the signature? I don't know for sure, but it does have an interesting consequence. You can overload methods based on this:

.assembly extern mscorlib { }
.assembly Test { }
.module Test.exe

.class public Program
{
  .method public static void Foo(int32[0...,0...] foo)
  {
    ret
  }

  .method public static void Foo(int32[1...,1...] foo)
  {
    ret
  }

  .method private static void Main()
  {
    .entrypoint
    ldc.i4.1
    ldc.i4.1
    newobj instance void int32[0...,0...]::.ctor(int32, int32)
    call void Program::Foo(int32[0...,0...])
    ret
  }
}

This is a valid and verifiable application. Unfortunately, using the  .NET reflection API it is impossible to see the difference in signatures between the two Foo methods.

This limitation in reflection means that code compiled with the new IKVM.Reflection.Emit based ikvmc won't be able to call methods that have non-zero-based array parameters. It is also impossible to override these methods, but that was already the case with the previous System.Reflection.Emit based implementation as well.

Finally, in the above text I talk about the lower bounds, but the same thing applies to the upper bounds of the array.

Monday, 17 November 2008 07:08:46 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]
# Friday, 14 November 2008
Introducing IKVM.Reflection.Emit

Over the past two months I've been working on reimplementing a large portion of the Reflection.Emit API from scratch. After finally growing tired of the System.Reflection.Emit bugs and limitations and not finding Mono.Cecil satisfactory either, I decided to build my own implementation.

I started out with these design goals:

  • API compatible with System.Reflection.Emit (as much as possible).
  • Implement only the Emit part and be compatible with System.Reflection.
  • Only implement functionality required by ikvmc, but not implemented functionality shouldn't silently fail (i.e. it should throw a NotSupportedException or NotImplementedException). This also means that the API is mostly write-only.
  • Efficient implementation, optimized for ikvmc specific scenarios.

I think I've met or exceeded all of the design goals. Without doing any significant performance work on my Ref.Emit implementation (other than the design), ikvmc became so much faster that it is rather emberassing for the Microsoft System.Reflection.Emit implementation.

I've only had to make a couple of minor changes to the ikvmc sources (apart from changing using System.Reflection.Emit; to using IKVM.Reflection.Emit; in every file) to account for the fact that IKVM.Reflection.Emit.ModuleBuilder and IKVM.Reflection.Emit.AssemblyBuilder unfortunately cannot extend System.Reflection.Module and System.Reflection.Assembly. However, it looks like this is fixed in the .NET 4.0 CTP.

Here are some random statistics about compiling IKVM.OpenJDK.ClassLibrary.dll on .NET 2.0 SP2 x64:

  System.Reflection.Emit     IKVM.Reflection.Emit  
File size 31,645,696 30,480,896   bytes
CPU time used 272 35   seconds
Peak virtual memory 1,433,399,296 1,035,018,240   bytes
Generation 0 GCs 770 896  
Generation 1 GCs 201 240  
Generation 2 GCs 11 8  

(The huge memory usage is not because it actually needs that much memory, but simply the result of the fact that garbage collection is more efficient if you have more memory available and that my system had about 1.5GB of free memory while running these tests.)

The smaller file size is because System.Reflection.Emit always uses fat method headers and IKVM.Reflection.Emit uses tiny method headers whenever possible.

There is still some work left to do, I've only spent limited time on debugging support and there is no support for Mono's .mdb format yet. I also haven't done any testing on Mono yet.

BTW, thanks to Sebastien Pouliot for code I lifted from Mono to parse strong name CAPI key blobs.

Other changes in this snapshot:

  • Dropped support for Visual Studio 2005.
  • Added error message when map.xml references non-existing constructor.
  • Added more statistics to ikvmc -time option output.

As always with a development snapshot, don't use this in production, but please do try it out and let me know about it. The sources are available in cvs and the binaries here: ikvmbin-0.39.3240.zip

Friday, 14 November 2008 10:01:50 (W. Europe Standard Time, UTC+01:00)  #    Comments [2]
# Wednesday, 12 November 2008
IKVM 0.38 Release Candidate 2

A codegen bug was found (not a regression, so there will be a 0.36 update as well) so here's another release candidate.

Changes:

  • Changed version to 0.38.0.2.
  • Fixed openjdk.build BOM issue on Linux.
  • Fixed jsr verifier bug that caused incorrect codegen under very specific circumstances (thanks to Brian Schwallier for tracking down a repro).

Binaries available here: ikvmbin-0.38.0.2.zip
Sources (+ binaries): Sources: ikvm-0.38.0.2.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip

Wednesday, 12 November 2008 06:13:29 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Wednesday, 05 November 2008
IKVM 0.36 Update 2 Release Candidate 2

As announced when 0.36 was released, I will periodically release updates to 0.36 as long as there is enough interest in .NET 1.1 support.

This is the second release candidate of the second update.

Changes (all are back ported fixes):

  • Changed version to 0.36.0.13.
  • Fixed ikvmc not to open the key file for write access.
  • Added more efficient float/double to/from int/long bits converters.
  • Fixed libikvm-native.so build to include reference to gmodule-2.0 library.
  • Fixed ikvmc not to open the key file for write access.
  • Fixed Graphics2D.rotate() to convert rotation angle from radians (Java) to degrees (.NET).
  • Applied awt patch #1979656 by Daniel Wilson.
  • Fixed three String bugs found by OpenJDK string tests.
  • Fixed ldc <class> where <class> is a ghost array.
  • Fixed bug in instanceof <class> where <class> is a Serializable[].
  • Removed incorrect DataFormatException thrown in java.util.zip.InflaterHuffmanTree.
  • Fixed #2001802 contributed by Andy Malakov.
  • Fixed #2001799.
  • Fixed #2006953.
  • Made finalize() and clone() methods in cli.System.Object and cli.System.Exception final.

Binaries available here: ikvmbin-0.36.0.13.zip
Sources (+ binaries): ikvm-0.36.0.13.zip

Wednesday, 05 November 2008 10:08:46 (W. Europe Standard Time, UTC+01:00)  #    Comments [0]
# Tuesday, 04 November 2008
IKVM 0.38 Release Candidate 1

A couple of minor fixes.

Changes since RC0:

  • Updated version to 0.38.0.1
  • Hide nested types generated by callerID support
  • Made finalize() and clone() methods in cli.System.Object and cli.System.Exception final
  • Fixed reflection to work on .NET 2.0 RTM

Note that even though I fixed reflection to work on .NET 2.0 RTM, it still isn't a supported platform, I strongly recommend .NET 2.0 SP1 or higher.

Binaries available here: ikvmbin-0.38.0.1.zip

Sources: ikvm-0.38.0.1.zip, classpath-0.95-stripped.zip, openjdk6-b12-stripped.zip

Tuesday, 04 November 2008 06:36:51 (W. Europe Standard Time, UTC+01:00)  #    Comments [1]