Code generation using reflection

A couple of month ago, in late November I think, I got an idea when I was riding my bike home from work: using code generation to optimize code that normally relies on reflection. (To understand this post, you should at least now the basics basics of dynamic proxies. Check this out if you don't.)

A common pattern for me is to have an interface and generating the class(es) that implements it at runtime using a Dynamic Proxy. The behavior of the class defined by code that is parameterize by annotations on the interface. Example:

interface CommandLineArguments {

@ArgumentName({"c", "config"})
String configFileName();

@Argument({"x", "max")
String maxValue();

@Argument({"n", "min"})
String minValue();
}

The annotations on the methods in the interface defines what they should do simply by giving the name of the corresponding command line switch. Extremely terse, readable, and flexible code. If you ask me, this (annotated interfaces + dynamic proxies) is one of the best thing in the Java language.

The code that actually gets executed when the methods in the interface is called often rely heavily on reflection. Reflection, as everyone know who have ever used it, is very powerful but can also be very slow. Is there some way to make it a bit faster? Yes, there certainly is: code generation.

Lets take a simple but realistic example: for any interface, create a wrapper that print the name of the method and delegates to some other class that implements the same interface. The code for doing this look something like this:

import java.lang.reflect.InvocationHandler;

class InvocationPrinter implements InvocationHandler {
private Object delegateTo;

InvocationPrinter(Object delegateTo) {
this.delegateTo = delegateTo;
}

Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
System.out.println(method.getName() + " called.");
return method.invoke(delegateTo, args);
}
}

This is a general but, unfortunately, slow. It is trivial to speed up, but this requires us to hand-write every method for every interface we wish to use this way. Another way, which gives the same speed-up, is to generate the same code dynamically. Literally the same code (except for indentation and such). Then using Javassist, this code can be compiled to a class at run-time, resulting in bytecode with the same performance as your hand-written code.

I have prototyped this approach for generating code by providing wrapper classes that looks and behaves just like java.lang.reflect.Method, java.lang.reflect.Constructor, etc, except that they also stores how they were used. For example, the class (called BIMethod) that corresponds to java.lang.reflect.Method stores the arguments used when invoking it and the returned object. By doing this you can write normal Java code that uses reflection (via these provided wrappers), but also generate (at run-time) the Java code that implements the same functionality. In fact, since the wrappers keep track of returned values, and created object (via Constructor.newInstance) it is possible to do fairly complex stuff like:

void doSomeReflectionStuff(Object[] args) {
DummyInterface obj = null;
for (final BIConstructor c : factory.constructors(Dummy.class)) {
try {
obj = (DummyInterface) c.newInstance(args[1], args[2]);
break;
} catch (final Exception e) {
}
}

final BIMethod someMethod = getSomeMethod();
return someMethod.invoke(obj, 0, args[0]);
}

That is, you reflectively invoke an object created using reflection. In addition, a constructor matching the arguments (the Object[] args) is found automatically by checking if the constructor threw an exception of not. The generated Java code for this will look something like this:

Dummy variable0 = new Dummy(arg1, arg2);
return variable0.theChosenMethod(0, arg0);

If you wish to take a peek at the prototype, just go ahead. Be aware, though, that is is probably the least tested stuff I'll written in quite a while... there a probably heaps of bugs... :) Despite this, I think it's worth taking a look at if you need to get more performance out of your reflection-based code. Please contact me if you have any questions or ideas.

It's 2009 and you can't read a forwarded mail

The other day, a college sent me a mail that I simply forwarded from Outlook to my Gmail. Today, when I finally had time to read it I opened it in Gmail. What do you think the mail contains? Nothing, except an attached file called smime.p7m. This file contains the encrypted mail, apparently, so I can't read it.

Oh, please! Come on! Why is a simple thing like this so hard?! Really... seriously, I'm failing to forwardning an e-mail...? Are we really making progress?

Yeah, I know that I should have forwarded it without encryption. But why is this something I need to know about? The mail client should tell me that the receiver won't be able to read the mail... It's freaking 2009! Not 1979!

Who knows... in 3009, perhaps we humans have evolved enough to have figured out and understand this whole send plain stupid text to another person-thingie. It's apparently too advanced to grasp for the current generation of humans.

(I have high hopes for the next-gen humans, though... No, really. I do!)

Contracts? Test-driven? Insight!

The other day I pair-programmed with a new guy at work. He showed my a class and its tests he and another guy had written a few week earlier. I don't recall exactly what the class did, but it was quite simple, hence the tests was short and simple. Overall well written tests if you ask me.

As we looked at the tests we had the following conversation, which afterwards gave me an new insight to an idea I've had a long while: tests are contracts.
He: Most of these tests are for testing that the class logs correctly...
Me: Yes, is that a problem?
He: Well, I know TDD says that you need to write a failing test before you're allowed to write any production code. But isn't testing logging a bit over-kill?
Me: I understand what you mean. What kind of logging is this? Why does the class need to log?
He: We needed it to understand the code. For debugging.
Me: Is the logging needed now? Is there some script that parses these logs, for example?
He: No, it's not needed any more.
Me: In that case I'd say that these tests isn't needed.

I could go even further and say that those tests shouldn't be written at all and should be removed. I actually think that tests like these are more confusing then anything else. I'm not saying those two guys who wrote these tests did anything wrong; they were doing TDD and was doing TDD right. What I'm saying is that, in my opinion, TDD isn't ideal.

Yeah, I hear you're cries: What?! Heresy! Calm down. I'll try to explain.

My opinion is that a class' tests should define what the class has to fulfill to be considered correct. To be precise, with 'correct' I mean 'what makes all things that depend on the class behave correctly'. (I realize that this is an recursive definition of 'correct', but you're a human being so you can handle it. :))

Recall that my pair-programming partner said that the logging wasn't needed any more. This means that we could remove the part of the code that logs and the class would still be correct according to the definition above. However, the class would not pass its tests because of the tests that tests the logging. This means that the class is over-specified. This is bad. The solution? Ditch the logging tests!

And so my fellow pair-programmer did.

I often say that test-methods should be named shouldFoo since it makes you focus on the behavior that is tested instead of the part that is tested (the tested method for example). I'm thinking of extending this tip to nameing test-methods as shouldFooBecauseBar. If this convention was followed, the the test-methods that tests the logging whould be named shouldLogBecauseWeNeedItforDebugging. That name sound a bit silly, doesn't it? That because it is silly to test it!

As I said, a class' tests defines what the class has to fulfill to be correct. In other words, the tests is the contract that the class must fulfill. Having tests that define contracts is much better, I think, than having tests for every little piece of functionality a class have (i.e., TDD). One reason is that it makes it easier to understand how you can change the class without breaking anything.

Now don't get me wrong, TDD is great, really great. But is it perfect? Of course not, that would be very naive to think (not to mention boring: Nothing more to do here, we've find the ideal solution! Now, let's drink tea all day!).

Is contracts ideal? Probably not. It contracts better? Yes, I think so.

Pythonic parsing and keeping corner-cases in the corners

I've been fiddeling with Python for a while, especially a nice library called Pyparsing. I have posted some stuff about parsers before, and I have tried ANTL for a private project for parsing and translating Java code. Anyway, Pyparsing has to be the most intuitive and easy to use parser library I have used.

In my opinion, a common problem with many libraries, programming languages, etc., is that they are not opted for the most common, simple, cases. Rather, they make the most common cases just as hard as the most weird corner-cases you can possibly think of. Take this Java code for reading an entire file into a String:

FileInputStream fstream = new FileInputStream("filename.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
StringBuffer content = new StringBuffer();
while ((strLine = br.readLine()) != null) {
content.append(strLine);
}
in.close();
return content.toString();

Why, oh why, do I have to write all this code when all I want is (in Python):

return open('filename.txt', 'r').read()

or (in Perl):

open FILE, "filename.txt";
$string = <file>;

The Java API for opening and reading files seems to be focused on covering all possible use-cases. Covering all use-cases is of course a good thing, but not on the expense of common simple cases. It is trivial for me to add a few convenience methods/classes to cover the common cases. But why aren't these methods/classes in the API from the beginning?

There are several other examples of this screw-the-common-cases-and-make-the-api-super-generic-mentality. Reflection in Java throws a gazillion exceptions, for example, and in most cases you don't need to know what went wrong, only that it did go wrong.

So, anyway, let's get back to the Pyparsing library. As I said, it is very easy to use and the common cases are straight-forward to implement. For example, there are helper classes/methods for parsing a string while ignoring up-/downcase, for matching one (and only one) of a set of grammar rules, etc. In addition to this the +, ^, | operators, etc, are overloaded so a grammar rule normally looks something like this:

greet = Word( alphas ) + "," + Word( alphas ) + "!"

Awesomeness.

So, what is this post all about? Pyparsing or bad libraries? Both. There are so many bad libraries out there that aren't ment to be used by human programmers. That is, the most simple things in the library are hard to use just becase the harders things are har to use. Pyparsing, on the other hand, is a joy to use. I was suprised how often I thought oh, it would be nice to have such-and-such helper function now and after looking in the Pyparsing documentation though yey, there it is!

One factory to bring them all, and at run-time bind them

Most of us know that to call new makes code hard to test because the piece of code calling new is statically bound to another class. In other words, there is no way to isolate the class under test such that its interactions with other classes can be tested. There is of course a solution to this: factories and/or dependency injection (with or without a framework).

I personly prefer to manually inject the needed dependencies, and seldomly use DI frameworks such as Guice. Why? Well, call my old fashioned, but I find it easier to read and understand non-DI framework code. So this leaves me with writing factories.

I don't know about you, but I find it boring to write these lousy factories (also, how can they be tested?). All they do is calling new; should there be some better way of doing it? Of course there should, and in fact, there are (for some definition of "better" :)). I've implemented a generic factory that can be used to instantiate most classes.

The factory uses some reflection magic and does a fair amount of work at run-time instad of compile-time. Actually, the factory is on the edge of being a primitive DI framework. :) As I usually do in my post I cut to the chase and give a few code examples:

final Factory factory = Factory.of(ConcreteClass.class, AnotherConcreteClass.class);

This creates a new factory that can create two concrete classes. Let's say that these classes implement the interfaces Interface and AnotherInterface respectiveley. To get an instance of the Interface the factory is called like this:

final Interface instance = factory.create(Interface.class);

When the create method is called, the factory finds the class that implements the provided interface (in this case this is the ConcreteClass class). Then the factory instantiates that class and returns the instance. This means that

1. the factory can be used for creating different kinds of objects as long as they implement different interfaces,
2. the client of the factory is unaware of the concrete class implementing the interface, and
   
3. the same Factory class can be used in test-cases testing the client code (the difference is only how the factory instance is created).

I illustrait the last bullet with an example, This code is part of a test-case testing a class which use the factory to create an Interface instance:

final Factory stubbedFactory = Factory.of(Stub.class);

where, of course, Stub implements Interface (compare to the example above where the factory is created using ConcreteClass and AnotherConcreteClass).

Ok, so far so good. But how to instantiate a class that need some parameters to be instantiated, i.e., only has a non-default constructor? Parameters that the factory needs in order to create objects are provided by calling the using method:

final DependencyUser user = factory.using(new Dependency()).create(DependencyUser.class);

Here, the concrete class implementing DependencyUser is instantiated with the Dependency instance as argument to the class' constructor. Any number of parameters can be given to the using method as long as they have different (run-time) type. The factory will use those parameters that are needed for instantiating the class that the client requests.

The using method returns a new Factory instance that "know about" the parameters provided to using (and any parameters known by the factory instance which using was called on). This may sound a bit strange, but it makes it possible to add parameters as they become available in the program flow. For instance, it's quite common that some parameters are only know when the factory is used. Example:

// Outside client code. An instance of 'Two' is not available.
final Factory factory = Factory.of(ClassUsingOneAndTwo.class).using(new One());

// Inside client code. An instance of 'Two' is now available.
final OneAndTwoUser user = factory.using(two).create(OneAndTwoUser.class);

(ClassUsingOneAndTwo is a class implementing the OneAndTwoUser interface and its constructor takes an One instance and a Two instance).

That is! That's the mostly simple, almost generic, factory I hacked together... pardon me ...developed this afternoon. The source is available here. There is also a few test-cases in that Eclipse project.

JUnit 4.5 in Eclipse 3.4

I was quite disappointed when I realized that Eclipse 3.4 (Ganymede) still uses JUnit 4.3. JUnit 4.4 was released last summer so it shouldn't be rocket science to it a part of Ganymede, which was released this summer. Well, I guess it all comes down to lack of time and resources. Of course, it could also be because of compatabilty issues between JUnit versions, or something similar.

Luckely, it quite easy to use a newer version of JUnit in Eclipse. By simply adding the new JUnit JAR to the classpath of an Eclipse project and removing the JUnit 4 Library (that is, the entry in 'Java Build Path' with an icon that looks like a pile of books), the project will run the test-case with the new JUnit. Of course, your other projects will still use JUnit 4.3. We have done this at work with JUnit 4.5 and it works really good.

I recommend you to do this, since there are some really nice features in JUnit 4.5; the possibility do distinguish between assumptions and assertions, and better descriptions of failed assertions, for instance.

Don't Repeat Yourself - what does 'repeat' really mean?

It is not uncommon to have some kind of script (e.g., a .sh-script) to start an application. For example, a start script for an Java application could check that the correct version of the JRE is installed, set up the classpath, and then start the application by executing java -cp [classpath] [mainclass].

In a case like this, the start script contains some information that is already embedded in the source code, e.g., the name of the class containing the static void main(String[]) method. Is this a violation of the DRY principle? I certainly think so.

However, you could argue that source code is filled with this kind of violation (refering to something by its textual name) since classes/types are referred to by name everywhere, for example when instantiating a new object in most OO languages. I don't consider this to be a violation of DRY, though.

Why? Because with modern IDEs classes can be renamed/moved and all references to the class will be updated. Thus, effectively, there is no repetition (since you don't manually handle it). So, no violation of the DRY principle.

However, if the application uses reflection, or something similar, then the IDE can't safely handle it. Consequently, you have to handle these repetions manually with IDE support. In other words, the DRY principle is violated.

The impact of these violations can be minimzed by having a good test-suit. This way, if you fail to update the code correctly the tests will tell you so. Reflection-heavy code is not different than any other code in this sense.

Ok, so let's get back to the original example: the start script and the reference to the main class of the application. This is a violation of the DRY principle since the IDE does not update the script's references to classes. But not only that, in most cases it does not have any test-cases. This is very bad, because you'll get no indication that something has gone wrong. (You could argue that you shouldn't rename the main class, but that's beside the point I'm making).

So, how to fix this? Simple. Either

1. unit test the start-script (run it, or use some kind of pattern matching), or
2. generate the test-script by a well-tested generator.

All executable parts of the application it should be possible to test; but how about the non-executable parts? How about documentation, e.g., user guides? I don't have a good answer to this besides generate what can be generated but this is hard in practice. If you have a good soluation, please let me know...