Lessons from a debugger

A few days ago I got undefined method `some_method' for nil:NilClass when I executed a test-case I've just had written for a quite well-tested class. The test-case tested input that the class hadn't been designed to handled, but now I needed the class to handle it.

Knowing that there was a suit of test-cases making sure I didn't break anything, I just added return if (!thing) (where thing is the object the some_method was called on) and run the suit again. And gues what? All test-passed -- including the one I just written that didn't pass. I wrote another one testing a similar scenario and it also passed. I was satisfied.

Why did I add that nil-check? Well, the error appeared deep in a recursive call-chain, and I simply guessed that the recursion should be stopped if thing was nil. I didn't know -- I just guessed.

The point is that I didn't have to know, because if I was wrong any of the class test or multi-class tests would tell me I was. This is all good right? Well, it's not all good. Why? I'll tell you in a moment.

But first, think about a hard problem that you solved by putting in more effort than usual -- persuading someone to test more, parsing a proprietary file format, writing C++ or reading Perl -- any hard problem will do. I'm pretty sure you learnt something really valueable from that experience (even though it didn't feel that way while solving it...). I'm sure we learn a lot from solving any problem by putting in more effort than ususal.

Now, back to my story about the nil fix that made the test-case pass. What did I learn from fixing it by adding a line that I simply guessed should be there? Zip, nothing, ingenting. Of course, this shows that well-tested code is a Good Thing, but this isn't news to anyone. (By the way, note that a Good Thing isn't trademarked, registered, closed-sourced, or anything like that. Its free to use and I encurage you to do so as often as possible. :) Of course, any improvements you make to a Good Thing have to be shared with the community.)

On the other hand, what would I have to do if the class was poorly tested? I would have to understand what the code did by reading, test/run it by hand, debug it, etc, before I added the nil-check. Then I would have to repeate the process to make sure everything worked as before. What would the outcome of all this work be? Probably the same nil-check as before, but I would also understand the code much better. Also, I might picked up some good design ideas, learnt to use the tools better, etc.

Now, I'm not saying that poorly tested code is good. What I am saying is that working with poorly tested code that forces you to fire up the debugger and step through the program will make you debug programs better. I'm saying that working with deep inheritage hiearchies will make you realize that inheritance isn't always a good thing. I'm also saying that reading code with a lot a mutable instance variables and class variables will make you appreciate (and use) 'final' or 'const' more.

To take this a bit further, I think you actually get worse at debugging if you developing in an environment where the code is well-tested, because you never have to debug anything. This is true for me, at least.

I've completely stopped using the debugger. I write test-cases that narrow down the problematic code instead. This combined with a few print-outs is all I need. I think this is easier, and more valuable in the long-run because my efforts are mirrored in a few test-cases that document that there was a bug that was fixed. Would I simply had fired up the debugger and found (and fixed) the problem, there would be no (exectubale) documentation of the bug-fix.

So, one way of becoming a better developer is, I think, to improving quality of untested code because it'll forces you to reason about the program and it's control flow based on scarce information (e.g., logs and stack-traces) among other things. On the other hand, working with well-tested code is much easier: write a test, make the change you have to make to the production code and run all tests. Do they still pass? Great! Code is ready to be checked-in. Good for the project's progress. What have you learnt? Nothing! Bad for you. In some sense.