Java initialization chaos

I've recently started using Java again after going from C++ to Java to C++ and now Java again. What bums me out the most with Java is how it pretty surface hides a lot of chaos. Don't get me wrong, it's good that you don't have to see the chaos (too often), but it's terrible that the chaos is there at all. Java's static initialization is particular "interesting" topic.

What is a and b in the following code:

class A { public static int a = B.b; }

class B { public static int b = A.a; }

I know its a contrived example, but non-obvious variants of this code pops up every now and then and brings havok to the dependency graph.

So, what do happen to a and b in the code above? How does the JVM resolve the cyclic dependency? It does it by initializing a and b to zero. In fact, all (static) fields are first initialized to zero (or null if the fields holds a reference type) when the JVM first sees the field during class loading. Then when the entire class is loaded, it's class initializer is called where the fields are initialized to the values written in the source code, e.g., 10 in the code below:

class C { public static int c = 10; }

In other words, the code for C is equivalent to:

class C {

  public static int c = 0;

  static { c = 10; }

}

Which you can easily see by running javap on the C's .class-file.

Now, what I've been asking my self is why it's done like this? Why is the fields initialized to zero (or null) and later assigned to their proper value (10 in the case of C.c). It would be extremely helpful to have the fields being tagged as uninitialized before a  proper value is assigned to them. That would not require tonnes of extra logic in the JVM, and would capture circular dependencies upon startup of the application.

The JVM is nice and all, but it's not perfect. But then again, no one is arguing it is, right?

std::fstream -- please leave!

During the last half year, there has been several bugs related to C++ file streams, e.g., std::ifstream. Most bugs have been resolved by adding calls to ios::clear to clear the error state of the stream, or similar fixes. Other bugs were fixed by using C FILE instead.

Why, why, why, is file I/O so hard to implement in a reliable way in C++? I haven't done much work with files in C, but the things I've done work well. Same thing with python. Java's file I/O is just a joke (why do I need three classes to read a file?), but at is more reliable than C++.

Think twice before I use std::fstream again. You code might be fine on implementation of the C++ standard library, but will fail on another. Sad.

Andersson's Law

Proebsting's law states compiler advances double computing power every 18 year --- a pretty depressing fact.

Another depressing fact is that the most used language appeared to the public in 1973 -- almost 40 years ago.

The second most used language is essentially a combination of language features developed in the 70th and 80th -- 30 to 40 years ago. This language appeared in 1995 -- 16 years ago.

The third most used language is 30 years old and is based on a 40 years old language with some added features developed 40 years ago.

And the list goes on... Here is a compilation of the ages of the top 10 most used programming languages:

1. 38 (C)
2. 16 (Java)
3. 28 (C++)
4. 16 (PHP)
5. 16 (JavaScript)
6. 20 (Python)
7. 10 (C#)
8. 24 (Perl)
9. 37 (SQL)
10. 16 (Ruby)

Of course, just because a language is old doesn't mean it bad. Not at all. On the contrary, an old language is proven to be good (in some regard).

What bothers me though, is the "new" languages, e.g., Java, C#, or Ruby, which don't really add any kind of innovation except new syntax and more libraries to learn. Come on, there are tonnes of more interesting problems to solve... There is still no way of automatically parallelize a sequential program for instance.

There seems to be a new law lurking in programming language development... I call it Andersson's Law: Modulo syntax, innovation in new programming languages approaches zero.

And here's the "proof":
Every year there are new programming languages, however, a wast majority of those are merely reiterations of features found in previous languages (except syntax). Thus, the number of unique features per new language approaches zero for each year, that is, innovation approaches zero.

Forgotten treasures I: Look-up tables

Think fast: how would you convert a bool to a string, "TRUE" or "FALSE", in C? Most of us probably answers "with an if-statement", and when asked to show some code doing it end up with something like:

const char* theString;
if (theBool)theString = "TRUE";
elsetheString = "FALSE";

or some similar code using the conditional operator as follows:

const char* theString = (theBool ? "TRUE" : "FALSE");

But there is at least one more way of doing it, which in some cases gives more readable code. I'm speaking of look-up tables. The code above can be expressed by look-up tables like this:

static const char* options[] = { "FALSE", "TRUE" };
const char* theString = options[theBool];

which is much less code compare to the above if-then-else version. It's slightly more code, though, compared to the conditional operator-version. However, I personally find code using the conditional operator hard to read. I know perfectly well how the conditional operator works, but I still need a few more brain cycles to read such code.

On the other hand, I have no problems at all reading code using look-up tables since code like options[theBool] looks very similar to a normal function call. Of course, there are down-sides to using look-up tables; the most important one is that the result is undefined if you index outside the range of the table.

Another way of using arrays like this is when you need to decode a tree structure. Let's say we have three ints which represents a path in a tree.
   Node root = { "Root", layer1Nodes[node1], 0 };
   Node layer1Nodes[] = { { "L1A", layer1ANodes[node1]  } };

During the last year I've gone from primarily developing Java to write C++, C, and assembler. I have to admit, there are a few idioms that I have rediscovered in this transition to C and assemble. Look-up tables are on such treasure.

Of course, there is now real reason why look-up tables can't be used in, e.g., Java, but it seems like the coding traditions in Java discourage such code. Other languages encourage such  code; for instance lists and dicts are very common in Python as a tool for simplifying expressions. It's a bit funny what different communities value in their code.

DSL: Domain Specific Logging

A few years ago, I was asked to to design a logging framework for our new Product X 2.0. Yeah, I know, "design a logging framework?". That's what I thought too. So I didn't really do much design; all I did was to say "let's use java.util.logging", and then I was done designing. Well kind of anyway...

I started to think back on my first impression of the logging framework in Product X 1.0. I remembered that I had a hard time understanding how it was (supposed) to be used from a developer's point-of-view. That is, when faced with a situation where I had to log, I could not easily answer:

• to which severity category does the entry belong (FINEST, FINER, FINE, INFO, WARNING, SEVERE, ERROR, etc)
• what data should be provided in the entry and how it should be formatted, and
• how log entries normally is expressed textually (common phrases, and other conventions used).

In other words, it was too general (i.e., complex) with too much freedom to define your own logging levels, priorities, etc.

That's why I proposed to make logging easier, thus, the logging framework in Product X 2.0 has very few severity levels. This is implemented by simply wrapping a java.util.logging.Logger in a new class that only has the most necessary methods.

This simplification reduced the code noise in the logger interface, which made it a lot easier to choose log-level when writing a new log message in the source. This had an important implication (that we originally didn't think of): given that a certain message had to be logged, we (the developers) agreed more on which level that message should be logged. No more, "oh, this is a log message written by Robert -- he always use high log level for unimportant messages".

Of course, you could still here comments like "oh, this log message is written by John -- he always use ':' instead of '=' when printing variables". This isn't much of an issue if an somewhat intelligent being is reading the logs, e.g., a human. However, if the logs are read by a computer program, this can be a big problem -- this was the case for Product X 2.0.

This variable-printing business could be solved quite easily; we simply added a method to the logger class (MyLogger) called MyLogger variable(final String name, final Object value) that logged a variable. Example:

logger.variable("result", compResult).low("Computation completed."); 

which would result in the following log message:

2008-apr-12 11:49:30 LOW: Compuation completed. [result = 37289.32];

When I did this, I began to think differently about log messages. It put me into a new mind-set -- the pattern matching mind-set. Patterns in log messages started to appear almost everywhere. Actually, most of our logs followed one of the following patterns:

• Started X
• Completed X
• Trying to X
• Failed to X
• Succeeded to X

Here is an example:

try {
  myLogger.tryingTo("connect to remote host").variable(
                    "address", remoteAddress).low();
  // Code for connecting.
  myLogger.succeededTo("connect to remote host").variable(
                       "address", remoteAddress).low();
} catch (final IOException e) {
  myLogger.failedTo("connect to remote host").cause(e).medium();
}

To take this even further, it is possible to only allow certain combinations of succeededTo, tryingTo, cause, etc, by declaring them in different interfaces. Let's assume that it should only be possible to use cause if the failedTo method have been called. Here's the code for doing that:

interface MyLogger {
  // ... other logging methods.
  CauseLogger failedTo(String msg);
} 

interface CauseLogger {
  LevelChooser cause(Throwable cause);
  LevelChooser cause(String message);
}

interface LevelChooser {
  void low();
  void medium();
  void high();
} 

This is essentially what's called fluent interfaces. It complicates the design of the logging framework, but also makes it impossible to misuse it. Good or bad? It's a matter of taste, I think, but it's a good design strategy to have up your sleeve.

Testing generated code

I started to write this post roughly one and a half years ago. I never finished writing it until now for whatever reason. Here's the post.

<one and a half years ago>
At work I'm currently devloping a little tool that generates Java code for decoding messages (deeply nested data structures) received over a network. To be more precise, the tool generates code that wraps a third-party library, which provides generic access to the messages' data structures. Slightly amusing is that the library consists of generated Java code.

The third-party library is... clumpsy to use, to say the least. Its basic problem is that is so generic/general that it's feels like programming in assembler when using it. Ok, I'm exaggerating a bit, but you get the point: its API is so general it fits no one.

There are several reasons for hiding a library like this:

• simplifynig the API such that it fits your purpose;
• removing messy boilerplate code that's hard to read, understand, and test;
• less code to write, debug, and maintain;
  
• introducing logging, range checks, improved exception handling, Javadoc, etc, is cheap;
• removing the direct dependency to the third-party library; and
• you get a warm fuzzy feeling knowing that 3 lines of trivial DSL code corresponds to something like 60-80 lines of messy Java code.

I'm developing the code generator using Ruby, which have had its pros and cons. The good thing with Ruby is that is easy to prototype things and ou can do fairly complex stuff really easy.

On the bad side is that I reqularly get confused about what types go where. This isn't to much of a problem from a bug point-of-view since test-cases catches the mistakes I make, but it's a bit of an annoyance if you're used to developing Java with Eclipse like I am. The Ruby IDE I'm using (Eclipse RDT) is not nearly as nice as Eclipse JDT, which is natural since an IDE for a dynamic language has less information available for refactorings, content assist, etc.

I've discovered that a functional style is really the way to go when writing code generators, especially when there are no requirements on performance. This is a nice fit, beacuse Ruby encurage a functional programming style -- or rather, it encurage me.

What keeps biting me when it come to code generators is how to test them. Sure, the intermediate steps are fairly easy to test, that is, while things are still objects and not just a chunk of characters. But how is the output tested?

Usually I test the output (the generated code) by matching it against a few regual expressions or similar, but this isn't a very good solutions as the test-cases are hard to read. Also, if an assertion fails the error message isn't very informative (e.g., <true> was not <false>). Furthermore, test-cases like these are still just an approximation how the code really should look like. For example, I've found no general and simple way of testing that all used classes (and no other classes) are imported. So, it is possible that a bug such as:
import tv.muppets.DanishChef;
wouldn't be found by my test-cases, even though the resulting code wouldn't even compile (do'h! The chef's from Sweden not Denmark!).

Ok, this could be fixed by having some test-cases that actually compiles the output by invoking a Java compiler. Not very beautiful but still possible. Even better, the generated-and-compiled code could be tested with a few hand-written test-cases. These test-cases would test the generated code, alright, but would not document the code at all (since "the code" here means the uby code that genreated the executed Java code). This problem, I'm sad to say, I think is impossible to solve with reasonable effort.

This approach is good and covers a lot of pitfalls. However, it misses one important aspect: generated documentation. The code generator I wrote generated Javadoc for all the methods and classes, but how should such documentation be tested? The generated documentation is definitely an important part of the output since it's basically the only human-readable part of it (in other words, the generated code that implements methods are really hard to read and understand).

Another approach is to simply compared the output with the output from a previous "Golden Run" that is known to be good. The problem here is, of course, to know what is 'good'. Also, when the Golden Run needs to be updated, the entire output of the (potential) new Golden Run has to be manually inspected to be sure it really is good/a Golden Run. The good thing with a "Golden Run" is that documentation in the generated code is also tested and not only the generated code.

The approach I'm using is to have a lot of simple test-cases that exercises the entire code generator (from input to output). Each of these test-cases verifies a certain aspect of the generated code, for instance:

• number of methods;
• number of public methods;
• name of the methods;
• a bunch of key code sequences exists in method implementations (e.g., foo.doIt(), new Bar(beer), baz.ooka(boom), etc);
• for a certain method the code looks exactly like a certain string (e.g., return (Property) ((SomethingImpl) s).propertyOf(obj););
• name of the class;
• name of the implemented interfaces;
• number of imports;
• that used classes are imported;
• key frases exist in Javadoc.

In addition to these test-cases, there is are test-cases that simply generates a lot of Java classes and compiles them. The test-cases pass if everything compiles. Finially, some of these generated classes are tested using hand-written JUnit test-cases. For me, this approach worked good and I didn't experience any problems (meaning more problem than normally) with testing the generated code. For the generated documentation, however, there were much more bugs. These bugs didn't get fixed either because it wasn't high priority to fix them. Too bad.
</one and a half years ago>

Looking back at this story now, I would have done things differently. First of all I would not use Ruby. Creating (internal) DSLs with Ruby is really easy and most of the time turn out really nice. I've tried doing similar things with Python, Java and C++, but the syntax of these languages just isn't as forgiving as Ruby's. However, the internal DSL I created for this tool never used the power of Ruby, so it just be an external DSL instead. An external DSL could just as well be implemented in some other language than Ruby.

Second, I would consider just generating helper methods instead of entire classes and packages of classes. So instead of doing:
void setValueFrom(Message m) {
this.value = new GeneratedMessageWrapper(m).fieldA().fieldB().value();
}
you would do
void setValueFrom(Message m) {
this.value = getValue(m);
}
@MessageDecoder("Message.fieldA.fieldB.value")
int getValue(Message m) {
}
where the code for the getValue method would be generated and inserted into the file when the class is built. This way, much less code would be needed to be generated, no DSL would be needed (annotations are used instead), and things like name collisions would not be such a big problem as it was (believe it or not).

At any rate, writing this tool was a really valuable experience for meany reasons that I wouldn't wish to have undone. Considering how good (meaning how much code it saved us from writing) it was a sucess. On the other hand, considering how much time we spent on getting it to work it was a huge failure. As it turns out, this tool has been replaced with a much simpler variant written entierly in Java. Although simpler, it gives the user much more value, though, so its arguable much better then that stuff I wrote 1,5 years ago. My mistake.

Code generation: C++ vs. Java

Code generation can be very useful by reducing the amount of code that is needed to be written, tested, debugged, maintained, etc. But if you develop the code generation yourself, code generation can be a nightmare. Getting the code generator right can be very hard, especialley so if the generator needs to support more than one target language (the output language).

The last code generator I wrote was for a private project. I've always wanted to design and implement my own programming language, and this spring I finally started to implement a compiler (actually it was source-to-source translator that outputted C++ code) for my language, which I call Fit. This was the first translator/code generator I've written that generates C++ code.

I'm not a big fan of C++, I think it's a too complex language that forces you to think about low-level implementation details. Thus, since my brain is limited, I have less brain-cycles to think about the high-level design of the application.

So why did I choose to translate Fit to C++ if C++ is so hard? Well, my other alternaive was Java, but since I already had written a couple of generator for Java I thought that a generator for C++ would be a fun experience.

Actually it was a really nice experience! Having access to all the low level details (e.g., pointer arithmitics and goto) and the high-level constructs (e.g., operator overloading) made it relatively easy to translate Fit to C++. When implementing the Fit-to-C++ translator I really saw the difference in power between C++ and Java.

Java, although a good language, is not a powerful language. In Java you can do certain very easy, but as soon you need to do things the language wasn't designed for you will have a bad experience. Ever tried to write bit manipulations in Java? It's not fun I tell you. In C++ at least you have the option to implement it using operator overloading and templates to make is less of a burden. I Java, that option is not available.

To take a concrete example from the Fit compiler/translator: yield is translated to (efficient) C++ using a switch and a goto. I'm not sure how to implement it in Java, but I guess you would have to do some work-around using a do-while construct.

So what am I'm saying? Is C++ a good language? Well, no it's not good if a human being writes it, but it is actually good if a code generator writes it. All the low-level stuff actually simplifies the generator.

Java-compatible syntax for C++

Since I first realized how much more productive you are in Java compared to C++, it has bugged me that the syntactic difference is so small. Take this code as an example:

class A {
public void a() { }
}

Is that C++ or Java? (Hint: add ":" and ";" and it becomes another language). The difference is syntactically tiny, but huge when you think of all the things you get from Eclipse when using Java.

So, this the idea: express C++ with a syntax that is compatible with Java. This Java-compatible syntax (JCS from now on), of course, requires a program to translate it to C++, but it will make it possible to use a number of tools currently only available for Java. Refactoring and code browser (which is reliable), for example.

Yeah, I hear you, "you can't express advanced-feature-X and meta-programming-feature-Y using a JCS". You're right; macros and advanced template-programming is far beyond the reach of a JCS, but that's not my point. My point is that the majority of C++ code could easily be expressed with a JCS. If 95% of your code could be refactored or browsed using Eclipse, thats much better than if 0% of your code could be refactored properly.

Actually, I think that the alot of the C++ language is an example of not keeping the corner-cases in the corner. Simple things like writing a script to list all defined functions in a source file is (in the general case) impossible because macros can possibly redefine the language... (thus all #include:ed header files has to be parsed, thus the entire build system with all its makefiles has to be known to the scrip.).

I know Bjarne Stroustrup had reasons for doing this (backwards compatability with C), but I think this was more of marketing reason than a technical reason. His new language could have been compatible with C (being able to call it, and be called from it, etc) without the new language having to be a syntactial superset of C. Anyway, back to JSC for C++.

Friends and colleagues have told me that the new CDT for Eclipse gives you refactoring, code completion, and browsing, but it works poorly from my experience. Perhaps I've failed to configure Eclipse correctly, or I'm using a crappy indexer to index my C++ code... but I can't refactor my C++ code the way I can with Java code. (Compare the number of available refactorings in Eclipse for Java and C++ if you like to have an objective measurement).

I've implemented a prototype that proves that it is possible to create a JCS that covers the most common part of C++. It works by traversing the Java AST (abstact syntax tree) and translates relevant nodes to its C++ representation. Example:

class A extends B implements C {
public int foo(@unsigned int[] a, boolean b) {
if (b) return a[1];
return 0;
}
}

translates to

class A : public B, public C {
public: virtual int foo(unsigned int* a, bool b) {
if (b) return a[1];
return 0;
}
};

There are very much that's not covered with this prototype, and it's probably riddled with bugs... but it fulfills its purpose perfectly: proving that expressing C++ using a JCS is possible. The prototype is available here.

I'd love to make a real-world worthy implementation of this idea, but I'm afraid it will take up my entire spare-time... I have other things to think about! :)

Movable proxy: a design pattern for hiding dirty secrets

The Proxy design pattern is a useful pattern. For my style of programming it's quite common, although I seldom actually name the proxy-class SomethingProxy. Instead I try to come up with a name for its actual role or responsibility. An example of this kind of proxy could be the services provided by a package in some Java code. All classes of the package is package private (default visibility) except for one, which is the proxy towards the package (there are of course a bunch of public interfaces).

Traditionally, though, a proxy is an object used to accessing something complex in a easier way. For instance a proxy for sending messages to, or retrieving the state of, a remote process. At work, where I'm working with a distributed Java application, we have a bunch of such proxies for accessing the various distributed subsystems/processes. All these proxies have some protocol for talking to the remote subsystem, e.g., homebrew protocol N, standard protocol M, etc.

This is all good, except for the fact that both the client subsystem and the server subsystem need to share a dirty secret: which protocol that is used to communicate. A sane design will hide this dirty secret inside some class (e.g., a proxy!), such that the bulk of the code doesn't need to know the secret. But still, some part of the client subsystem must know it, otherwise it cannot establish a connection to the server.

Or does it?

Wouldn't it be great if it was possible for the client to ask the server for the proxy instance? That is instead of doing:

final Proxy forTalkingToServer = new ServerProxy();

the client does:

final Proxy forTalkingToServer = server.proxyForTalkingToServer();

Ok, that sound like a nice idea. But wait, how is the proxyForTalkingToServer() method implemented? Doesn't the implementation of that method need to communicate with the server? Well, yes. But this bootstrap problem is easily solved by having a standardized protocol for sending java objects between different Java systems, e.g., RMI.

The sequence is something like this:

      Client                      Server
      ------                      ------
        |    Proxy request [RMI]    |
        |-------------------------->|
        |                           |
        |    Proxy confirm [RMI]    |
        |<--------------------------|
        |                           |
+------------------+                |
| client has proxy |                |
+------------------+                |
        |                           |
        |    Whatever request [?]   |
        |-------------------------->|
        |                           |
        |    Whatever confirm [?]   |
        |<--------------------------| 

Where Proxy request is what happens when the proxyForTalkingToServer() method is called, and Proxy confirm is the methods' return value, that is, the proxy the client should use for talking with the server. The Whatever request and Whatever confirm messages are messages sent using some protocol (symbolized by [?]) that the server decided to use.

In addition to having the client completely unaware of the actual protocol used for communicating with the server, the proxy can be pass around (using, e.g., RMI) the entire distributed system and it will still be possible to access the server using the proxy. It is also possible to update the protocol without updating the clients, or to let the proxy instance implement multiple interfaces (one new-and-improved and one legacy for backwards compatibility). It is also great for testability/debugging: you can easily replace a troublesome protocol with a simpler variant.

Neat.

Code generation using reflection

A couple of month ago, in late November I think, I got an idea when I was riding my bike home from work: using code generation to optimize code that normally relies on reflection. (To understand this post, you should at least now the basics basics of dynamic proxies. Check this out if you don't.)

A common pattern for me is to have an interface and generating the class(es) that implements it at runtime using a Dynamic Proxy. The behavior of the class defined by code that is parameterize by annotations on the interface. Example:

interface CommandLineArguments {

@ArgumentName({"c", "config"})
String configFileName();

@Argument({"x", "max")
String maxValue();

@Argument({"n", "min"})
String minValue();
}

The annotations on the methods in the interface defines what they should do simply by giving the name of the corresponding command line switch. Extremely terse, readable, and flexible code. If you ask me, this (annotated interfaces + dynamic proxies) is one of the best thing in the Java language.

The code that actually gets executed when the methods in the interface is called often rely heavily on reflection. Reflection, as everyone know who have ever used it, is very powerful but can also be very slow. Is there some way to make it a bit faster? Yes, there certainly is: code generation.

Lets take a simple but realistic example: for any interface, create a wrapper that print the name of the method and delegates to some other class that implements the same interface. The code for doing this look something like this:

import java.lang.reflect.InvocationHandler;

class InvocationPrinter implements InvocationHandler {
private Object delegateTo;

InvocationPrinter(Object delegateTo) {
this.delegateTo = delegateTo;
}

Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
System.out.println(method.getName() + " called.");
return method.invoke(delegateTo, args);
}
}

This is a general but, unfortunately, slow. It is trivial to speed up, but this requires us to hand-write every method for every interface we wish to use this way. Another way, which gives the same speed-up, is to generate the same code dynamically. Literally the same code (except for indentation and such). Then using Javassist, this code can be compiled to a class at run-time, resulting in bytecode with the same performance as your hand-written code.

I have prototyped this approach for generating code by providing wrapper classes that looks and behaves just like java.lang.reflect.Method, java.lang.reflect.Constructor, etc, except that they also stores how they were used. For example, the class (called BIMethod) that corresponds to java.lang.reflect.Method stores the arguments used when invoking it and the returned object. By doing this you can write normal Java code that uses reflection (via these provided wrappers), but also generate (at run-time) the Java code that implements the same functionality. In fact, since the wrappers keep track of returned values, and created object (via Constructor.newInstance) it is possible to do fairly complex stuff like:

void doSomeReflectionStuff(Object[] args) {
DummyInterface obj = null;
for (final BIConstructor c : factory.constructors(Dummy.class)) {
try {
obj = (DummyInterface) c.newInstance(args[1], args[2]);
break;
} catch (final Exception e) {
}
}

final BIMethod someMethod = getSomeMethod();
return someMethod.invoke(obj, 0, args[0]);
}

That is, you reflectively invoke an object created using reflection. In addition, a constructor matching the arguments (the Object[] args) is found automatically by checking if the constructor threw an exception of not. The generated Java code for this will look something like this:

Dummy variable0 = new Dummy(arg1, arg2);
return variable0.theChosenMethod(0, arg0);

If you wish to take a peek at the prototype, just go ahead. Be aware, though, that is is probably the least tested stuff I'll written in quite a while... there a probably heaps of bugs... :) Despite this, I think it's worth taking a look at if you need to get more performance out of your reflection-based code. Please contact me if you have any questions or ideas.

Pythonic parsing and keeping corner-cases in the corners

I've been fiddeling with Python for a while, especially a nice library called Pyparsing. I have posted some stuff about parsers before, and I have tried ANTL for a private project for parsing and translating Java code. Anyway, Pyparsing has to be the most intuitive and easy to use parser library I have used.

In my opinion, a common problem with many libraries, programming languages, etc., is that they are not opted for the most common, simple, cases. Rather, they make the most common cases just as hard as the most weird corner-cases you can possibly think of. Take this Java code for reading an entire file into a String:

FileInputStream fstream = new FileInputStream("filename.txt");
DataInputStream in = new DataInputStream(fstream);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
String strLine;
StringBuffer content = new StringBuffer();
while ((strLine = br.readLine()) != null) {
content.append(strLine);
}
in.close();
return content.toString();

Why, oh why, do I have to write all this code when all I want is (in Python):

return open('filename.txt', 'r').read()

or (in Perl):

open FILE, "filename.txt";
$string = <file>;

The Java API for opening and reading files seems to be focused on covering all possible use-cases. Covering all use-cases is of course a good thing, but not on the expense of common simple cases. It is trivial for me to add a few convenience methods/classes to cover the common cases. But why aren't these methods/classes in the API from the beginning?

There are several other examples of this screw-the-common-cases-and-make-the-api-super-generic-mentality. Reflection in Java throws a gazillion exceptions, for example, and in most cases you don't need to know what went wrong, only that it did go wrong.

So, anyway, let's get back to the Pyparsing library. As I said, it is very easy to use and the common cases are straight-forward to implement. For example, there are helper classes/methods for parsing a string while ignoring up-/downcase, for matching one (and only one) of a set of grammar rules, etc. In addition to this the +, ^, | operators, etc, are overloaded so a grammar rule normally looks something like this:

greet = Word( alphas ) + "," + Word( alphas ) + "!"

Awesomeness.

So, what is this post all about? Pyparsing or bad libraries? Both. There are so many bad libraries out there that aren't ment to be used by human programmers. That is, the most simple things in the library are hard to use just becase the harders things are har to use. Pyparsing, on the other hand, is a joy to use. I was suprised how often I thought oh, it would be nice to have such-and-such helper function now and after looking in the Pyparsing documentation though yey, there it is!

One factory to bring them all, and at run-time bind them

Most of us know that to call new makes code hard to test because the piece of code calling new is statically bound to another class. In other words, there is no way to isolate the class under test such that its interactions with other classes can be tested. There is of course a solution to this: factories and/or dependency injection (with or without a framework).

I personly prefer to manually inject the needed dependencies, and seldomly use DI frameworks such as Guice. Why? Well, call my old fashioned, but I find it easier to read and understand non-DI framework code. So this leaves me with writing factories.

I don't know about you, but I find it boring to write these lousy factories (also, how can they be tested?). All they do is calling new; should there be some better way of doing it? Of course there should, and in fact, there are (for some definition of "better" :)). I've implemented a generic factory that can be used to instantiate most classes.

The factory uses some reflection magic and does a fair amount of work at run-time instad of compile-time. Actually, the factory is on the edge of being a primitive DI framework. :) As I usually do in my post I cut to the chase and give a few code examples:

final Factory factory = Factory.of(ConcreteClass.class, AnotherConcreteClass.class);

This creates a new factory that can create two concrete classes. Let's say that these classes implement the interfaces Interface and AnotherInterface respectiveley. To get an instance of the Interface the factory is called like this:

final Interface instance = factory.create(Interface.class);

When the create method is called, the factory finds the class that implements the provided interface (in this case this is the ConcreteClass class). Then the factory instantiates that class and returns the instance. This means that

1. the factory can be used for creating different kinds of objects as long as they implement different interfaces,
2. the client of the factory is unaware of the concrete class implementing the interface, and
   
3. the same Factory class can be used in test-cases testing the client code (the difference is only how the factory instance is created).

I illustrait the last bullet with an example, This code is part of a test-case testing a class which use the factory to create an Interface instance:

final Factory stubbedFactory = Factory.of(Stub.class);

where, of course, Stub implements Interface (compare to the example above where the factory is created using ConcreteClass and AnotherConcreteClass).

Ok, so far so good. But how to instantiate a class that need some parameters to be instantiated, i.e., only has a non-default constructor? Parameters that the factory needs in order to create objects are provided by calling the using method:

final DependencyUser user = factory.using(new Dependency()).create(DependencyUser.class);

Here, the concrete class implementing DependencyUser is instantiated with the Dependency instance as argument to the class' constructor. Any number of parameters can be given to the using method as long as they have different (run-time) type. The factory will use those parameters that are needed for instantiating the class that the client requests.

The using method returns a new Factory instance that "know about" the parameters provided to using (and any parameters known by the factory instance which using was called on). This may sound a bit strange, but it makes it possible to add parameters as they become available in the program flow. For instance, it's quite common that some parameters are only know when the factory is used. Example:

// Outside client code. An instance of 'Two' is not available.
final Factory factory = Factory.of(ClassUsingOneAndTwo.class).using(new One());

// Inside client code. An instance of 'Two' is now available.
final OneAndTwoUser user = factory.using(two).create(OneAndTwoUser.class);

(ClassUsingOneAndTwo is a class implementing the OneAndTwoUser interface and its constructor takes an One instance and a Two instance).

That is! That's the mostly simple, almost generic, factory I hacked together... pardon me ...developed this afternoon. The source is available here. There is also a few test-cases in that Eclipse project.

JUnit 4.5 in Eclipse 3.4

I was quite disappointed when I realized that Eclipse 3.4 (Ganymede) still uses JUnit 4.3. JUnit 4.4 was released last summer so it shouldn't be rocket science to it a part of Ganymede, which was released this summer. Well, I guess it all comes down to lack of time and resources. Of course, it could also be because of compatabilty issues between JUnit versions, or something similar.

Luckely, it quite easy to use a newer version of JUnit in Eclipse. By simply adding the new JUnit JAR to the classpath of an Eclipse project and removing the JUnit 4 Library (that is, the entry in 'Java Build Path' with an icon that looks like a pile of books), the project will run the test-case with the new JUnit. Of course, your other projects will still use JUnit 4.3. We have done this at work with JUnit 4.5 and it works really good.

I recommend you to do this, since there are some really nice features in JUnit 4.5; the possibility do distinguish between assumptions and assertions, and better descriptions of failed assertions, for instance.

A run-time equivalent to JUnit's @Ignore

It's been some time since I wrote about things that annoy me. Now it's time again. The pain-in-the-lower-back this time is: why isn't there a (good) way to ignore a JUnit test-case based on a piece of information that only awailable at run-time? In short: dynamically ignore a test-case.
I think a "good way" to solve this should fulfill the following:

• a dynamically ignore test-case should marked as "ignored" in the JUnit test-run,
• possible search for dynamically ignored test-cases in the IDE.

There is a very simple way to ignore a test-case base on run-time information:
@Test
public void testIt() {
  if (shouldIgnore())
    return;
  // ... the rest of the test-case.
}
However, this solution does not fulfill the above requirements at all. Something better is needed.

JUnit uses a org.junit.runner.Runner to run test-cases. Since JUnit 4.0 it's possible to define which such Runner that should be used to run a set of test-cases. The @RunWith annotation does just this. Here is an example:
@RunWith(MyRunner.class)
public final class MyTest {
  // ... some test-cases.
}
There are several ways @RunWith can make you testing filled days easier; for a real-world example you need to look no further than to JMock. I have implemented a Runner that makes it possible to do:
@RunWith(RuntimeIgnoreable.class)
public final class MyTest {
  @Test
  public void perhapsIgnored() {
    ignoreIf(perhapsTrue())
    // ... the rest of the test-case.
  }
}
Neat, ey? I think so at least. What's even neater is that the RuntimeIgnoreable class and the ignoreIf method was embaressingly straight-forward to implement. You can browse the code, or look at the individuals files:

• RuntimeIgnoreable
• ignoreIf
• example

Oh, one final note: this was develop for JUnit 4.3. If you are using any other JUnit version, the you prbably need to make some minor changes to the code.

Update: I just realize that JUnit Extensions does this (among aother things)...

Checked exceptions exposes implementation?

I've read and heard the phrase Checked exceptions are evil! They expose implementation details! a couple of time (last time was an entartaining read for several other reasons...). I really don't understand this statement (hey, you, explain to me please).

How can the EncodingException part of the code below expose implementation details when void encode() does not? The first say I can fail to encode, the latter say I can encode. What's the difference?
interface Encoder {
  void encode(Object o) throws EncodingException;
}
For me, checked exceptions are vital to enforce proper error handling. But this can (probably) only be achieved if the exceptions fits the problem domain. For instance, throwing an IOException in the interface above would be really really bad because it exposes details of the Encoder, e.g., that is uses the network, or whatever. On the other hand, the only thing EncodingException exposes is that an Encoder can fail to encode the provided object. That's not an implementation detail, I think, that's the Encoder being honest and not hiding it flaws.

One of the most important lesson I've learnt from using exceptions is the importance of doing try-catch-wrap-throw. For example, if an implementation of Encoder uses a method that throws IOException then the proper way of handling such exception is to catch it, wrap it in a EncodingException and throw it to the client of the Encoder. This cleans up ugly interfaces that throws many exceptions, resulting in clear description (with semantics, yeay!) of what can go wrong when a method is called. Exceptions that fits the problem domain is the key.

A couple of times I've let domain exceptions inherit from each other to define that, for example, a RangeException is a ConfigurationException. This seemed like a good idea to me at the time, however, it seldom helped the design (it didn't make it worse either, though). In fact, the only time I find it useful is when you need to distinguish between thrown exceptions in one place but handle them in the same way in a nother place. For example (where LeftException and RightException inherits from BaseException):

interface Something {
  void doIt() throws LeftException, RightException;
}

class HandlesExceptionsSeparately {
  void method(Something s) {
    try {
      s.doIt()
    } catch(LeftException e) {
      // handle it
    } catch (RightException e) {
      // handle in another way) {
    }
  }
}

class HandlesExceptionsTogether {
  void method(Something s) {
    try {
      s.doIt()
    } catch(BaseException e) {
      // handles both
    }
  }
}

But as I said, this is not very common for me. Although if you are developing a library and wish the user to have the ability to handle the different error-cases separetely, then could be useful.

Well, these are some of my thoughts on checked exceptions. In summary: I think they're good stuff if done well. :)

LISP's mapcar for Java: onAll-collect

I've been working on a project were I need to iterate over a set of objects, get some property from all objects in the collection, and store that property in a new collection. This wouldn't be much if an issue if I did it in LISP or Ruby or some think similar... but this is done in Java with th the java.util.Collection framework. The Collection framework is nice and all, but when I have to write:


List<SomeProperty> allProperties(List<SomeClass> objects) {
  List<SomeProperty> properties = new ArrayList<SomeProperty>();
  for (SomeClass c : objects)
    properties.add(c.getProperty());
  return properties;
}
when all I really wish to say is:

(defun all-propreties (objects)
  (mapcar get-property objects))
I get a bit sad. Sure computer technology is progressing, but are computer languages? LISP appeared in the late 50-ies and Java in the mid 90-ties, but looking at the code above I can't really tell that there is almost 40 years between these two languages. Crazy...

Anyway, to make me a bit happier I started to think about how the Java code above could be improved to make it a little bit more terse. What I came up with was this:


List<SomeProperty> allProperties(Bag<SomeClass> objects) {
  return objects.collect(objects.onAll().getProperty());
}
which I honestly think is pretty neat. Then again, I'm just an ape-descended life form who are so amazingly primitive that I still think digital watches are a pretty neat idea too.

How does this work? Well, first of all the objects variable is not the same type in the two Java examples above. In the latter example, it is a class that I've implemented specially to deal with the scenario described. This class, which is called Bag<T>, has a method onAll(T) that returns an dynamic proxy implementing the T interface. That is, in the example above the objects.onAll() returns an instance of the SomeProperty interface.

This dynamic proxy handles every method call it receives by calling that method on each object in the Bag. Also, if the called method is non-void, then it returns whatever the last object in the Bag returned. This means that bag.onAll().someMethod() behaves just as thing.someMethod() if bag contains just the thing objects. This is a Good Thing in my book. :)

How about the Bag.collect method?, you ask. Funny you should ask that. I was just getting to that, I reply. Cut to the chase already!, you say. Cool it, I say, or there won't be any desert!.

(Ok, I'm getting a bit carried away.)

The Bag.collect simply returns the set of return values got when last calling a method on the onAll-object. Ehm, it a bit hard explain with words... but I think you understand. If you don't, look at the code and test-cases. :)



Note that the onAll method returns an object that can be used for more than what's described above. Whenever you need to treat a set of objects as if they were one object, for instance the Observer pattern, the onAll-object simplifies alot.

What makes a class?

A few days ago, when I was "merrily" washing my dishes, I started to think about what makes a class necessary. That is, how we could identify code that should be placed in a class. I came up with a few obvious cases, and some less obivous. Obvious:

• a new domain entity is needed,
• to simplify testing,
• remove conditional logic and special cases,
  
• remove duplicated code,
• move private private methods of a class to a new class; and
• split a class that has multiple responsibilities.

Yeah, I know, ob-vio-us... But as I said, when I joyfully made my forks and spoons clean and shiny, I found a couple of cases that are commonly ignored or missed (at least by me, until now):

• there are recurring patterns in variable names and variable type, e.g., java.nio.ByteBuffer payload = ... indicates that there is a need for a Payload class,
• circular dependencies between classes can be solved by introducing a new class, e.g., if Alice and Bob has references to each other for exchanging messages, then a Channel should be introduced that they both will use to send and receive messages;
• an exception exposing implementation details should be replaced with a new exception mathing the abstraction level of the class/method throwing the exception,

A comment on classes that solves circular dependencies: it is possible to solve this kind of dependency by simply introducing an interface (e.g., Alice and Bob could implement a MessageReceiver interface), but this misses the point a wish to make. By introducing a new class instead of an interface, domain logic can be placed where it is more suitable (e.g., the Channel can take care of serializing the messages passes between Alice and Bob).

Anyway, these are a few ways we can use classes to simplify code by introducing additional classes. There are probably a lot more... :)

Oh, by the way: some exceptions in the packages java.... miss a SomeException(String description, Throwable cause) constructor. Why is this? It makes catch-wrap-rethrow really hard. This is very important for hiding imlementation details.

Furthermore, why are there no proper hiearchy for (example for) the exceptions thrown by the reflection mechanism? I hate to catch four-five exceptions every time I do something with reflection... NoSuchMethodException, SecurityException, NoSuchFieldException, InvocationTargetException?! Come on! Why can't I just catch ReflectionException and get it over with? Also, give me multi-catch so that I can catch more than one Exception in each catch-block. Please!

Well, who ever said Java was perfect? Or even close to perfect...

Boring stuff you have to implement: Configuration, part 2

A while ago I wrote a post where I proposed an easy way of specifying the configuration of an application. The idea is basically to define a configuration parameter by annotating a methods with information that describes the parameter. Of course, the value of the parameter is retrieved by calling the annotated method. My previous post contains an example.

To implement this easy-configuration-thingie I use dynamic proxies. If you haven't heard of dynamic proxies you have missed one of Javas powerful facilities for metaprogramming. Under the circumstances (e.g., static type-checking) I think it's pretty easy to use too.

The basic idea behind dynamic proxies is quite simple: let all calls to methods of an interface be delegated to another method. This method is called invoke and is declared in java.lang.reflect.InvocationHandler.

As you may suspect, the invoke method receives the all arguements given to the method defined in the interface (i.e., the method that delegated to invoke). It also receives an arguments that describes which method that was called; this is an java.lang.reflect.Method object, which among other things, contains the method's annotations.

Back to the original topic: configuration. How can all this annotation stuff and proxy fluff be used to define and read configuration?

Well, as the example in my earlier post shows, the interface that defines the configuration is annotated with the name and the type of the configuration parameter. Since the method's annotations are available to the invoke method, invoke can use the parameter name to look up its value (in a hashmap, or similarly) and return it. It's as simple as that!

I've made a simple implementation of this availble here (follow the instructions on Google Code if you wish to check-out the entire Eclipse project).
Note that some more development is needed before this code is useful, since it does not
read any configuration from file (only default values can be read). 

In general, I tend to think that annotations simply are additional arguments to the annotated method (although a bit harder to use than ordinary arguments). This way of looking at annotations is even more suitable when used together with dynamic proxies, I think.

You probably already have thought of this, but there are several other ways of using annotations + dynamic proxies: I've used it to parse binary messages and command line arguments (before I know about JewelCLI), and I guess you can come up with several other examples...

Making deactivated logging 100 times faster

I think the java.util.logging is a nice logging framework: it's easy to do simple things yet it is not limited. You can easily tweak it using custom filters, formatters, and handlers. One thing I do not like with it, however, is its performance.
 
The problem with logging
I have no problem with the performance of java.lang.logging when the logging is activated. It's the performance when logging is deactivated that is an issue for me. The problem, as I see it, is that when
  logger.info("someting: " + something.toString());
is executed, the argument to info is created (by concatinating two string, which is computationally heavy) despite logging being deactivated. This means that a string will be created and then directly thrown away without being used. To make things even worse, there is even a greater performance penalty if the toString method of something is computionally heavy.

This is not problem with java.util.logging per se, but rather a problem with the Java language. Don't get me wrong -- I like Java -- but in certain areas Java is simply too limited/limiting. I see at least three way of solving the problem described above:

1. introducing some kind of macros to the language,
2. using aspect-oriented programming, or
3. performing string concatination lazily.

Personly, I think that the common variant of macros (the C/C++-kind) it a Bad Thing. On the other hand, the other variant of macros (the Lisp-kind) does not fit nicely in the Java languange because those kinds of macros operate on the AST (this is perfectly ok in Lisp because Lisp does not have any syntax -- you're actually creating the AST when you write the program).

The second solution to the problem is aspect-oriented programming. To be honest, I don't know enough about that to be able to discuss it here. With the limited knowledge I do have, however, I think that it should be possible to instrument the piece of code above such that you get the following sematic:
  if (logger.logsAtLevel(Level.INFO) {
    logger.info("someting: " + something.toString());
  }

The third solution -- performing string concatination lazily -- is the solution I will discuss for the rest of this post. I'm assuming that the methods used to create the log message, e.g., the toString method, are are pure functions, i.e., has no side-effect. This is a perfectly legitimate assumption because deactivated logging should have no side effect as it is.

Lazy string concatination
Ok, so how can we make string concatination in Java lazy? In C++ we could have overloaded the operator +, but this is not possible in Java. One hacker-ish solution would be to implement new String and StringBuilder classes (which the compiler uses to implement string concatination) which performs concatination lazily, but this not trivial... (I have actually tried... (and failed)). Instead, we can implement a thin wrapper around java.util.logging.Logger with the following methods:
  MyLogger log(Object msg);
  MyLogger log(Object msg1, Object msg2);
  MyLogger log(Object msg1, Object msg2, Object msg3);
  // ... and so on.
  void info(Object msg);
  // ... and all the the other levels.
which is used like this:
  myLogger.log("Received message: ", msg, " from ").info(msgProvider);
which is the equivalent of
  logger.info("Received message: " + msg + " from " + msgProvider);
when using a java.util.logging.Logger. The log methods is simply implemented by storing the references to the objects given as arguments. The info method is implemented by calling toString on its argument and the argument given to log if the logging is actived, otherwise it does nothing.

I (kind of) have implemented such class; the difference is that instead of wrapping a java.util.logging.Logger my class uses a java.util.logging.Handler directly. The interface of this class, which I named Ln4j (pun definitely intended), is the same as MyLogger above, however.

Performance measuments
So, what kinds of performance numbers can we expect? <disclamer>I'm definitely no expert in measuring performance, but I have tried my best to create fair benchmarks.</disclamer> These are the benchmarks:

• logging single constant string,
  
• concatinating two constant string and log the result,
• concatinating a constant string and a variable string and log the result,
• concatinating six short (4 characters) variable strings,
• concatinating six long (40 characters) variable strings,
• concatinating a constant string and an int and log the result,
• concatinating a constant string and a List<double> (of length 8) and log the result.

I ran these benchmars with and without the -server switch to the JVM and with logging activated and with logging deactivated. This is the result.

In summary: with logging activated ln4j performs a bit faster than java.util.logging. However, since ln4j is quite simple (e.g., it has no log levels) this small performance advantage would probably disapper if ln4j implemented all functionality provided by java.util.logging.Logger.
When running the benchmarks with logging deactivated, there is usually considerable performance gains (no, the post title is no exaggeration). Of course, exact numbers depend on what is logged. When logging a single constant ln4j is actually somewhat slower. However, in the benchmark that logs a list, ln4j is 600-700 times faster than java.util.logging. That optimzation for ya!

I hope this post was informative and that you have learned something from reading it. I learned a lot when experimenting with lazy string concatination; let's hope it will native in Java 8. :)

Oh, I almost forgot, here and here are the source used in the benchmarks.

Making MBean names first-class

Time for yet another problem that have annoyed me: names of MBeans. First of all, I find the something:key=value-notation noisy and non-intuiative in comparison to the dot-notation normally used in Java. This is, however, something I have gotten used to and have accepted.

What I have not accepted is that the MBeans are mere java.lang.String, which, to use an understatement, is not good because it forces developers to keep track on naming conventions, etc.

So, how to solve this? Easy, let's make MBean names first-class. This way, IDEs will help developers by suggesting possible keys and values in MBean name. Also, refactoring tools can be used to rename key and values, etc. Great stuff, I say!

Using some annotation tricks and reflection, I've made it possible to annotate an MBean with a special kind of annotation, which makes it is possible to do:

@something(key=value)
final class MyBeanImpl implements MyBean {
  // Code goes here.
}
which means that the name of MyBean is something:key=value. My current implementation takes an annotated class and returns the distinguised name; continuing the the example above you whould do like this to get the name of MyMbeanImpl:

final String myName =
  new DistinguishedName(MyBeanImpl.class).name();

I'm sure my implementation of this needs to be improved, but the concept is implemented by this class (see test-case for documentation), and this is how the @something looks like (well not quite, the linked code has different name, keys, etc, but I'll think you get it anyway).