Lambda the Ultimate Joker

From Lambda the Ultimate Weblog:

All people are lazy, but very eager when evaluated.

Funny stuff for all us language nerds. :)

Testing generated code

I started to write this post roughly one and a half years ago. I never finished writing it until now for whatever reason. Here's the post.

<one and a half years ago>
At work I'm currently devloping a little tool that generates Java code for decoding messages (deeply nested data structures) received over a network. To be more precise, the tool generates code that wraps a third-party library, which provides generic access to the messages' data structures. Slightly amusing is that the library consists of generated Java code.

The third-party library is... clumpsy to use, to say the least. Its basic problem is that is so generic/general that it's feels like programming in assembler when using it. Ok, I'm exaggerating a bit, but you get the point: its API is so general it fits no one.

There are several reasons for hiding a library like this:

• simplifynig the API such that it fits your purpose;
• removing messy boilerplate code that's hard to read, understand, and test;
• less code to write, debug, and maintain;
  
• introducing logging, range checks, improved exception handling, Javadoc, etc, is cheap;
• removing the direct dependency to the third-party library; and
• you get a warm fuzzy feeling knowing that 3 lines of trivial DSL code corresponds to something like 60-80 lines of messy Java code.

I'm developing the code generator using Ruby, which have had its pros and cons. The good thing with Ruby is that is easy to prototype things and ou can do fairly complex stuff really easy.

On the bad side is that I reqularly get confused about what types go where. This isn't to much of a problem from a bug point-of-view since test-cases catches the mistakes I make, but it's a bit of an annoyance if you're used to developing Java with Eclipse like I am. The Ruby IDE I'm using (Eclipse RDT) is not nearly as nice as Eclipse JDT, which is natural since an IDE for a dynamic language has less information available for refactorings, content assist, etc.

I've discovered that a functional style is really the way to go when writing code generators, especially when there are no requirements on performance. This is a nice fit, beacuse Ruby encurage a functional programming style -- or rather, it encurage me.

What keeps biting me when it come to code generators is how to test them. Sure, the intermediate steps are fairly easy to test, that is, while things are still objects and not just a chunk of characters. But how is the output tested?

Usually I test the output (the generated code) by matching it against a few regual expressions or similar, but this isn't a very good solutions as the test-cases are hard to read. Also, if an assertion fails the error message isn't very informative (e.g., <true> was not <false>). Furthermore, test-cases like these are still just an approximation how the code really should look like. For example, I've found no general and simple way of testing that all used classes (and no other classes) are imported. So, it is possible that a bug such as:
import tv.muppets.DanishChef;
wouldn't be found by my test-cases, even though the resulting code wouldn't even compile (do'h! The chef's from Sweden not Denmark!).

Ok, this could be fixed by having some test-cases that actually compiles the output by invoking a Java compiler. Not very beautiful but still possible. Even better, the generated-and-compiled code could be tested with a few hand-written test-cases. These test-cases would test the generated code, alright, but would not document the code at all (since "the code" here means the uby code that genreated the executed Java code). This problem, I'm sad to say, I think is impossible to solve with reasonable effort.

This approach is good and covers a lot of pitfalls. However, it misses one important aspect: generated documentation. The code generator I wrote generated Javadoc for all the methods and classes, but how should such documentation be tested? The generated documentation is definitely an important part of the output since it's basically the only human-readable part of it (in other words, the generated code that implements methods are really hard to read and understand).

Another approach is to simply compared the output with the output from a previous "Golden Run" that is known to be good. The problem here is, of course, to know what is 'good'. Also, when the Golden Run needs to be updated, the entire output of the (potential) new Golden Run has to be manually inspected to be sure it really is good/a Golden Run. The good thing with a "Golden Run" is that documentation in the generated code is also tested and not only the generated code.

The approach I'm using is to have a lot of simple test-cases that exercises the entire code generator (from input to output). Each of these test-cases verifies a certain aspect of the generated code, for instance:

• number of methods;
• number of public methods;
• name of the methods;
• a bunch of key code sequences exists in method implementations (e.g., foo.doIt(), new Bar(beer), baz.ooka(boom), etc);
• for a certain method the code looks exactly like a certain string (e.g., return (Property) ((SomethingImpl) s).propertyOf(obj););
• name of the class;
• name of the implemented interfaces;
• number of imports;
• that used classes are imported;
• key frases exist in Javadoc.

In addition to these test-cases, there is are test-cases that simply generates a lot of Java classes and compiles them. The test-cases pass if everything compiles. Finially, some of these generated classes are tested using hand-written JUnit test-cases. For me, this approach worked good and I didn't experience any problems (meaning more problem than normally) with testing the generated code. For the generated documentation, however, there were much more bugs. These bugs didn't get fixed either because it wasn't high priority to fix them. Too bad.
</one and a half years ago>

Looking back at this story now, I would have done things differently. First of all I would not use Ruby. Creating (internal) DSLs with Ruby is really easy and most of the time turn out really nice. I've tried doing similar things with Python, Java and C++, but the syntax of these languages just isn't as forgiving as Ruby's. However, the internal DSL I created for this tool never used the power of Ruby, so it just be an external DSL instead. An external DSL could just as well be implemented in some other language than Ruby.

Second, I would consider just generating helper methods instead of entire classes and packages of classes. So instead of doing:
void setValueFrom(Message m) {
this.value = new GeneratedMessageWrapper(m).fieldA().fieldB().value();
}
you would do
void setValueFrom(Message m) {
this.value = getValue(m);
}
@MessageDecoder("Message.fieldA.fieldB.value")
int getValue(Message m) {
}
where the code for the getValue method would be generated and inserted into the file when the class is built. This way, much less code would be needed to be generated, no DSL would be needed (annotations are used instead), and things like name collisions would not be such a big problem as it was (believe it or not).

At any rate, writing this tool was a really valuable experience for meany reasons that I wouldn't wish to have undone. Considering how good (meaning how much code it saved us from writing) it was a sucess. On the other hand, considering how much time we spent on getting it to work it was a huge failure. As it turns out, this tool has been replaced with a much simpler variant written entierly in Java. Although simpler, it gives the user much more value, though, so its arguable much better then that stuff I wrote 1,5 years ago. My mistake.

Happy two year!

One year ago this blog celebrated its first birthday. And today, one year later, it celebrates its second birthday! Gasp! Oh, arithmitic, you gotta love it.

During the last year I've written about some crazy stuff, some testing stuff, a lot of compiler stuff, a few things about learning, and even a joke!

I hope some of these posts have been useful for you and that you continue to stop by every now and then to read some of them. Now I'm looking forward to another year of crazy, and (I hope) few useful, ideas and hacks.

Designing (your)Self

I just saw a very interesting lecture by Davig Unger. Its main focus is the programming language Self, which is a prototyped-based object-oriented language. He also talks about other things: getting the right people to work with you, some of his guiding principles, and even a few comments on getting to know yourself.

David Ungar received the Dahl-Nygaard Senior Prize in 2009 for his work in the field of object-orientation, most notably for Self. He currenlty works at IBM Research.

Commodore 64 Live!

Do you remember those old arcade machines where, when no one played them, the "computer" would play the games? Kind of like a demonstration of the game instead showing some fancy graphics. I came across the übergeeky page C64 Longplays a few days ago and I feel that I must do my small part to spread the word through the world.

It's basic like those computer-played arcade machines, with the difference that it's actually someone who plays through the entire game. Didn't you solve the third level of Boulder Dash or did you think no one ever cared to finish Ghostbusters? Everything is on C64 Longplays for you to watch.

I personally liked the video of Yie Ar Kung Fu... some fights was really close... the energy bar was down to the last pixel!

The most important skill

As a software developer you need to have large skill set: knowing the right tools, languages, frameworks, platforms, and so on. To complicate things you need to keep every piece of knowledge updated since the field of computing is changing so rapidly. But what is the most important skill among all these? I argue that it's not knowledge of some key technology that make you more than an average programmer, nor is it how many fixed point combinators you dreamt up, how many open-source project you've started, or how fast you type. If non of these, what is the most important skill?


First let me note that with developer I do not mean someone like Alice who write shell one-liners for her husband Bob to search his photo collection of dogs with hats. I don't mean teenanger Charlie who uses barbed wire and duct tape to hack together scripts to play suitably random mp3s to match his parents' photo slide show from their vacation last summer. What I mean is someone who write code that other developers will maintain, test, read, enhance, etc. In essens I mean developers that work with other developers either directly or through their code.

So, the important part here is work with other developers, which leads me to the heart of this post: the most important skill for a developer is communication. Err, I'm sorry, let me correct that: the most important skill for every developer is effective communication. By communication I mean source code and comments, documentation and specifications, power-point presentations and white-board explanations, mail and phone calls, and so on. I could continue, but I think you've got the point.

A small disclaimer before we continue. I assume that the ones you communicate with want to understand what you say or write. If that's not the case then you have an entierly different problem, and I don't think much in this post can help you. Sorry. Ok, that's that, let's continue.

I've met many very technically skilled developers that, when I tell them I don't understand what you mean, literally repeated what they just said (only louder). Is this good communication? I don't think so. I've also got mails from otherwise good programmers which basically required mind reading ability in order to be understood. Even worse, the mails were sent to 10-50 people, meaning (10 to 50) * (mail length) / (reading speed) * (average speed reduction factor due to lack of mind reading ability) man-hours was spent reading that mail. Effective communication? No.

I think there are two kinds of communication: one-way and two-way. Explaning some intricate technical detail to you pair-programming collaegue is a two-way communication since you (should) instantly realize if you collaegue understands you (by her facial expressions and body language). Phone calls are also two-way since the vocal intonation (should) indicate directly to you whether or not your explanation is good enough.

On the other hand, I consider mail to be a one-way communication (especially if sent to many people) since you don't get any indications when you write the mail if it will be understood or not when received. The same is true for documentation, specifications, code comments and source code. All these kinds of communication is one-way since it takes long time until you get any feedback on if your colleagues understand it or not.

So, how to become a better developer? How to communicate more effectively? For two-way communication I suggest to practice. For one-way communication I suggest... practice, as well actually. That is. Move along, nothing more to see here. Hit the back-button on you favourite browser.

Just kidding. Well, kind of any way. I do think that practice is the one thing will make any noticable difference, so my suggestion is to try to explain whatever technical work you're currently doing to your aunt Mathilda who lives in that small red cottage and knits hats all day long. Oh, it's impossible you say? Your aunt has never heard of numerical stability? She thinks monte carlo simulations has something to do with SimCity? That's doesn't mean you can't make her understand the ideas behind your work. It only means that you have to try harder to explain and simplify it using her language and concepts! Remember that the whole idea with this exercise is to make you better at explaning to other people, not make your aunt capable of challenging the next Turing award winner. To be blunt, she is just an exercise tool. A excerise tool that makes extremely good blueberry pies, though. Yum.

In fact, you could do this exercise by explaning anything. For instance, describe the story of the last book you read, a news article, or the story behind the best photo you ever took. But remember, you're not just telling the story, you're explaning what happend. You should "listen" (with your ears and eyes) to your audience to make sure they understand every word you say.

Some simple concrete tips:

• Be clear about the background. Just because you remember the context, does not mean every one else do. Start the explanation by giving the relevant context. If you have discussed it before with the audience, refer to that. If you haven't, start by giving a hand-waving overview of the problem.
• Use you hands. If you talk about many different objects, place them on different places in the air and "hold" the object you talk about by making a fist on that spot. Much less confusing for those who listen to you. Also, if you need to move your arms around to refer to different objects, you will probably speak more slowly which is also makes it easier to take in everything you say.
• Pop the stack explicitly. Often in a discussion you need to get back to something you talked about earlier. In that case, say "as I said before" before you start talking about it to indicate to your audience that you pop the conversation stack. If you just change the subject people could get a bit confused and miss some parts of your explanation.
• Be utterly clear when something is important. Say "this is the key point", or "this is important". Use short sentences. Repeat yourself. Use short sentences and repeat yourself. Make a pause to let your audience take in what you just said and let them think about it. Use some uncommon phrase or a silly name when explaning this key point that you can use later on to refer to it.
• Speak clearly or stay quiet. If you don't know what to say, don't mumble something incomprehensible. Just shut up. Scratch your head a bit if you feel akward to let the audience know that you're thinking about what to say. People won't mind anyway, unless you do it twice every minute. They will probably thank you for the moment of mental relief. In fact, you can make pauses like this work for your advantage. (Read the bullet above one more time if you don't understand why. (You still don't get it? I'm refering to the 6th sentence, silly.))

That it for this post. But what about the one-way communications? How to improve those? I'll get back to that in my next post on this subject. Until then, if you have any other tips please let me know by leaving a comment!

And oh, before I forget, say hello to aunt Mathilda for me!

How browsers work

A former colleague sent me this very sweet explanation of how browsers work today. Can you say awww?