Thoughts on "The Clean Coder" by Robert Martin

I recently finished The Clean Coder by Robert (Uncle Bob) Martin, and frankly, I loved it. I've been a fan since I first read the Newkirk and Martin book on XP. Most of Uncle Bob's books deal with the technical issues of writing good code. The Clean Coder is a departure. Rather than talking about how to code, it takes on what it means to be a "Professional".

I love the book because it is unflinchingly honest, even when that is painful. The chapter on Saying "No" is the hardest read, not because it is poorly written but because it hits so close to home. The basic message is that, as professionals, we have an obligation to say "no" and mean it when confronted with unreasonable demands. Our usual response of "I'll try" is, basically, a lie. It's not a lie in that we won't try, but rather in the fact that we know good and well there is no way its going to happen; but we go along to avoid conflict. Business makes plans based on what they think are real commitments, only to be let down one more time.

Anyway, as I read this I reflected on the the past couple of projects I led that were, ahem, less than stellar successes. A couple of instances:

• Toward the beginning of a project, we got a handle on the general scope and functionality. I very painstakingly worked out an estimate of 18 months. This figure came out of 3 disparate methods, including one from a piece of commercial estimation software. Given that the 3 results were very similar, I thought I had done a good job. First meeting with my boss, I presented the results and was told, "that will never fly. It has to go operational within the next 9 months." They countered that they would add resources (guess what position is still open a year and a half later?). "I'll try", followed by a hellish year and my eventual resignation after the project scope, re-envisioned for the n-th time, was still in free-fall.
• Well into another project (different employer) I suddenly had a project manager foisted upon me. As I was leading development, managing staff, doing budgets and the PM role, this should have been a gift. Instead, I saw it as a turf grab, and we went through a couple of months of jousting before we came to an understanding and started working collaboratively. Development pretty much stopped for a few weeks while I pulled together detailed project history and a raft of estimates. The PM dumped it into MS project and we finally came up with a defensible estimate of completion. He was sure that having the details documented would show management the true state and we could all move forward to a quality conclusion ("kum-bai-ya"). Instead, this was immediately met with a rousing yawn followed by "9 months won't do. Version 1 has to be out this year" (this was September). My response was, of course, "I'll try". Three months of stress later, we pushed out a very lame, buggy version. Because the feature set was weak and it had a tendency to crash, it was a "non-successful" launch, and eventually funding was slashed for the next year, dooming the effort to the list of projects no one talks about. Oh yes, the contractor PM who I came to like and respect, was terminated less than a week before Christmas.

In looking at these, incidents, I have a three responses:

• to cringe at the lack of success
• a bit of PTSD (Post-traumatic Stress Disorder) recalling the feelings
• a serious desire to never go through this again.

I had pretty much blamed the people who uttered the "That won't do". But I've reading "The Clean Coder," I have had to reconsider. In retrospect, "I'll try" avoided short term conflict, at the expense of long-term project failure and personal stress.

Honestly, if your auto mechanic said it will cost $200. for new brakes and you told him "That won't do; make it work for $10 using cardboard and chewing gum", you (and your insurance company) would not be happy if his response was "I'll try".

I'm left with the conclusion the, once again, Uncle Bob is right. Saying "No" would have caused tremendous conflict. That conflict was with people who controlled my raises, and even my continued employment. And the statements of "That won't do" seemed to have thinly disguised threat of unemployment behind it. Nonetheless, I am convinced that the Professional response is not "I'll try." Having the guts to stand behind my view, to express the reasons and not cave to the pressure, is Professional. "I'll try" is a disservice me and to my customer; by getting along now, "I'll try" hoses everyone in the long term.

As I said, the book is great because it is unflinchingly honest, even when that is painful.

What the "-1.#QNB"? Debugging We Go

I recently compiled the code I have been writing about and ran a regression data set (aka an old operational run). The original had been run on a Linux box using Lahey's Fortran compiler, and we also had an older Solaris version.

Aside from the expected differences (end of line, time and date stamps, CPU time, etc), I ran across the text in the title of this post. Specifically, an average of some reasonable parameters was coming out:

"-1.#QNB" instead of "0.1327"

WTF?

The rest of the run looked fine, and the final estimates of the parameters were all correct.

I googled the "#QNB", and got a number of hits for the Qatar National Bank, Google not paying attention to the "#" at the beginning.

Adding the "1." yielded something useful. I found an old discussion thread at "www.rhinocerous.net" (apparently a site about parallel computing). The actual page is gone, but I found a cached version of the thread. Long story short, this is a "not a number" (NaN) message.

Both I and the original poster were using MinGW. Specifically, I'm using g77. The number in question was printed out using F8.4. Apparently cygwin does show this as NaN. It turns out so will MinGW, but only if there are enough digits of precision. When I print the quantity in question using WRITE(*,*), I get "-1.#QNAN". With only 4 decimal places, however, it appears that this rounds to "-1.#QNB". I never would have guessed.

For the record, I tried the varying the format from F8.0 to F10.8, and here's what I found:

F8.0 -1.

F8.1 -1.$

F8.2 -1.#R

F8.3 -1.#QO

F8.4 -1.#QNB

F8.5 -1.#QNAN

F10.6 -1.#QNAN0

F10.7 -1.#QNAN00

F10.8 ************

which makes sense, if you think of rounding the integer equivalent of the ASCII codes. Presumably more precision will simply yield more trailing 0's.

The F8.0 result is concerning, because in that case there is no indication that there is anything wrong.

And people wonder why I mumble when they ask me what I did all day.

A Short-cut for Comparisons

Having shown the lowest possible tech version of getting the comparison of files done, I thought I'd show a couple of simple improvements. These (and a few others) can be found in MS_DOS Batch Files, (2nd ed) by Kris Jamsa.

Instead of using dir and then editing the results, we can have DOS do the iteration over files for us. The basic syntax is:

FOR %%I IN (a, b, c) DO command

where command is any DOS command. Specifically, we can use it to call out comparison batch file "comp.bat".

Within the parentheses can be an arbitrary set of files. Wildcards are allowed, so we can write:

compall.bat:

FOR %%I IN (*.f) DO call comp

This one-liner replaces the 60-line file we created with Excel yesterday, and gives identical function.

Dude, where's my code? Sorting out a "wealth of files"

So, I've been given the job of doing a major clean-up on a small/medium-sized system (~12K lines of FORTRAN 77) that implements a core business functionality. The system is about 15-20 years old, and consists of sixty odd files. No tests for anything, and the bulk was written by statisticians. UGH.

I'm documenting the steps in the process as an aid to others (or at least an outlet for my whining).

Figure Out Where the Code Is

Historically, use of version control has been spotty in certain areas, despite my impassioned pleas. I discovered a secret to getting others to adopt it, however. I had a couple of high priority bug fixes within a month. When we were doing impact analysis, I got to say "Since we don't do version control, we have no idea when this happened. It could have been last release, or it could have been when the original version was written in 1992." Suddenly, every one thinks Subversion is a truly excellent idea.

Figure Out Which Versions to Merge

A survey of the department turned up two major versions of the source. Moreover, one of the major versions had spawned at least 4 minor variants. In addition, I'm guilty of having pulled part of the source down and made minor fixes. Each time, an emergency overcame the work and so I have several directories on my hard-drive with names like "temp4" and "PRE_FIX_07_2010". Any moral high ground that I had previously laid claim to just left the building. Sigh.

So, job 1 is to clean out all of the redundancy without losing any important enhancements/fixes. For this first pass, I'm taking the lowest tech approach possible. I could go faster by writing Python/Ruby scripts, but I wanted to keep tight control on this first pass. Automating would make the job faster, but I needed the confidence that comes with direct, hands on work.

Low-tech Automation: Batch Files

I chose one recent version as "base", and wrote two simple DOS batch files to do the comparisons (we live in XP world).

The first compares two files with the same name:

COMP.BAT

fc %1 C:\BASE_VERSION\*.*

fc is a simple file byte-by-byte file comparison. In the present case, it does the job because the contents are identical for about 80% of the files any two versions.

Next, I created a list of the files in the source directory:

dir /B > filenames.txt

The /B switch tells dir to report only file names (one per line), without the date of last modification, etc. The output is redirected into the file "filenames.txt". Using ">" overwrites the file if it exists; ">>" would append the results.

Now I pulled the filenames into Excel. I inserted a column before the column of names. I put "call comp" in the first cell, and then dragged it down in front of all of the file names. Similarly, I put ">> results.out" in the column after the filename and dragged the fill cursor down to copy it into each row. Recall that ">>" would appends the results to the file specified. Finally, I copied it to the clipboard, pasted it into an editor, and saved it to compall.bat.

The first few rows of the final file looked like:

call comp aaa.f >> results.out

call comp bbb.f >> results.out

etc.

Now all I had to do was copy both and into each directory, type compall, and open up results.txt. Files that compared the same could simply be deleted, allowing me to focus only on the differences.

Three things to remember about this approach

1) It is about as low tech as it gets.

2) I have to remember to delete the file results.out if I decide to re-run the job. Otherwise the new run is appended to the results of the original, which can be confusing if you haven't had your morning coffee.

3) Notice that I used "call comp" instead of just "comp". In a DOS batch file, "comp" transfers control to comp.bat program, which would compare the first file and then quit. I use "call comp"control returns to the original program (compall.bat) after comp.bat runs. Essentially, "call comp" is calling a function/subroutine/procedure, while the plain "comp" is a GOTO.

On to ... LISP (?)

My financee is a former high school math teacher. One of the term she's introduced me to is TYNT, this year's new thing, and it's companion, LYNT, last year's new thing. Apparently we aren't the only ones subject to fashions. After the CORBA, JINI, SOAP, SOA, SaaS, and the Perl/Python/Ruby/Groovy scripting wars, I'm a bit shell-shocked.

However, in spite of myself, I ave decided to come to grips with reality. Multi-core is here to stay, and the and now we have to figure out how to respond. The big question to me is which way to jump in response. We have our old threading tools (locks, semaphores,et al.), the new thread constructs in C++ standard library, the Intel tool set, and others.

To me, the most interesting development is is the sudden increase in prominence of functional languages. So, I'm taking a look at hybrids like Scala, F# and old favorites like Haskell, and a new incarnation of Lisp, Clojure. To warm up my functional muscles, I'm using it as an excuse to experience a blast from the past, and starting with The Little Schemer. Currently, I'm working through a chapter a night. The complete reliance on recursion seems a bit forced at times, but it is a good mental workout--yoga for the recursive part of the brain.

Agile in a Ossified Culture

A friend of mine (call him Fred) is trying to get a project approved in his company.  The point of the project is to replace a complicated, multi-step manual process with an automated tool.  In addition to being manual, the process relies heavily on tacit knowledge (i.e., it exists only in the users' heads).  The approach being suggested was agile: several short iterations geared toward delivering immediate business value while beginning to draw the tacit knowledge out of the users' heads.

To give you an idea of how well it is going, Fred is currently in month five of preparing/revising/defending the project plan.  He is hoping to go forward for actual approval in another six weeks.  Assuming all goes well, after budget approvals, etc. are done, the first line of code may be written in another eight weeks (OK, I lied; Fred got so frustrated that he's been doing some coding during this protracted farce).  Honestly, if the waterfall group got together with the PMP certification folks and the most conservative accountants and CMMI/ISO9002 group you could imagine, the resulting process would be cavalier in comparison.  There are a minimum of 16 meetings to review the final plan and budgets before it can be approved.

What is hard to understand is that everyone in Fred's company seems to be terrified by the prospect that version 1.0 won't contain every possible bit of detail, kind of like Athena appearing fully formed from the head of Zeus (although Zeus got whacked in the head with an axe to let Athena out, a feeling Fred is becoming quite familiar with).  What's more, they seem completely focused on having the perfect plan.  Remember this is a manual process that revolves around tacit knowledge.  One of the company's technology leaders asked how the data integrity of the results could be guaranteed.  Currently the "data integrity of the results" is based upon rules that only exist in people's heads, and on manual entry of the results into a back-end system.

Fred's company has a plan-driven culture.  The company essentially believes that planning predicts the outcome.  Requiring detailed plans is their way of preventing wild, poorly thought out wastes of the company's money.  Fair enough.  What I find puzzling it that they can't see that they are losing all of the business value that could be had now if they would just get out of their own way  Given that they seem to agree that the project should and will be done, opportunity cost associated with the delay seems like a foreign idea.

Agile development is often described as a strategy of mitigating risk of poorly understood and/or volatile requirements.  It is well-known that requirements churn have blown many a budget, and eventually killed many projects.  One thing that seems less emphasized is Agile's focus on early delivery of the highest priority requirements.  This can allow the business to realize some value from the software early, even while the plan-driven folks are still in the requirements huddle.

Fundamentally, I'm left wondering whether it is possible for Fred to succeed with his approach.  One possibility is obviously faking a detailed plan, and then doing he intended to do all along.  As long as the results are good, this can be done.

But if Fred wants to play it straight, can he?  How does one sell that to management that doesn't want to hear it?  Is it possible to use Agile be effectively used in a plan-driven culture such as exists in Fred's company?

Well-read? Isn't that special ...

This is an up front apology (of sorts).

This is intended as a forum for my thoughts about computer languages, software development, testing, etc. (and occasional gratuitious references to my kids and cats). A number of those thoughts are stolen from others, but I do try to give credit where credit is due.

I wandered into the software field later in life, and so I don't have a degree in computer science. Being self-taught leaves me with some feelings of inferiority, and I deal with mine through reading. I'm at a point where I can afford to buy books, and I spend much of my free time reading computer books. This does get me very strange looks at the beach on vacation (at least I think that's what the strange looks are for), but I'll put up with it. Anyway, my time in school left me with the tendency to cite things, and I intend to keep that up here. I'll try to keep the cites relevant (Brian Kernighan, P. J. Plauger, Bjarne Stroustrup, Guido van Rossum, and Joel Spolsky), rather than spinning off into that "As Pliney the Younger said..." kind of nonsense.

End of apology