Software Rhythm: Mid-Game

This is part two of the Software Rhythm series.

The Mid-Game is where the rubber hits the road. Your primary activities will be writing code and writing tests. I’ll let you decide which order to write them in, as long as you write them both.

Cycles

However, this series isn’t really about writing code per se but about the rhythm and commonalities of software development. Here in the mid-game we see cycles buried in cycles: releases contain iterations contain daily builds contain commit sets contain code/test/refactor cycles. ¹

I touched briefly on iterations in the Opening. Iterations are all the rage these days in the agile world. I’ve done development both with and without them and honestly I haven’t found that using iterations had a strong correlation with successful development. What I have seen is the value of visible public feedback during development. One way to do that is with iterations that provide working features on a regular basis. Another way is to do periodic milestone demos or usability tests or performance testing or whatever makes sense for your project.

The most important demo is the first one as you’re probably building something completely wrong and when the user or product manager sees it, they’re going to freak out. That’s ok. In fact, it’s great because you will get plenty of feedback to build the next version. ²

</p>

Here’s some feedback for you:

</center>

Feedback loops

Feedback loops are good. In fact, I’d say they are possibly the most important tool we have in fine-tuning the process of buliding software. This should not be surprising given that feedback loops are in general good tools for regulating complex systems.

If you notice all those cycles I listed earlier, it’s useful to think about how we can adjust what we’re doing in each trip through the cycle. I think you’ll find that many of our best practices happen to correlate to cycle feedback. Releases get feedback through acceptance testing and actual users. Iterations (or milestones) get feedback by showing work earlier to users and re-planning. Daily builds give feedback by running a suite of tests and reporting on quality. Daily standup meetings give feedback on who’s doing what. Commit sets give feedback by exposing your changes to others and possibly by running commit-time hooks for test suites. Code/test/refactor cycles give you feedback by green bar unit test runs. Post-mortems (or the trendier term “retrospectives”) can give the team feedback about itself at whatever frequency you want.

I’ve come to believe that most process improvements can be tied to a feedback cycle: either by creating a new one, tightening an existing one, or making the feedback easier to see and act on in the next cycle. If you’re casting about for ways to improve your processes, examine thyself with an eye towards feedback and see what you find.

Version control

The three most important things to know about version control systems:

If you’re not using version control, you’re doing it wrong.
If you’re not using version control, you’re doing it wrong.
If you’re not using version control, you’re doing it wrong.

I have spent extensive periods of time in my development life using RCS, VSS, MKS, Perforce, CVS, and SVN and while these differ wildly in usage and capabilities, they are still more similar than they are different. Whatever system you are using, the key is to know how to do these three things:

Get all changes from others into your local workspace
See how your local changes vary from the shared view
Push changes from your local workspace to the shared view

You should know how to do these things cold, preferably both from a command line AND from a tool/IDE. Getting changes from others inevitably involves understanding how to “merge” someone else’s changes into your local view. I think we’re finally starting to reach the point where merging is more science than art. However, I find it’s always safest to approach merging with the same caution as riding a motorcycle with a lion in the side car.

Once you’ve mastered this level, proceed to the next level by working with branches. It’s worth the effort. Being able to hack off a copy of the code, make changes, and pull it back in with reasonable confidence that you haven’t destroyed the system is quite liberating.

Builds

So, you’ve got this code. How do you take all those files (so many many files) and end up with an Application? Here we enter the mystical art of the build system where even Gandalf would wonder “Why the Frodo am I downloading the Internet“?

Your project needs a reliable, repeatable way to take the files under source control and create the thing the user uses, whether that’s a web app or something that gets sent on an actual physical disk to someone. You should test this process by running it at least every night. Even better, use continuous integration and run it all the time. (Note: tightening feedback cycle.)

I’m not going to get into specific build systems here but yours should be able to: retrieve the code from source control, compile the code, run tests, collect/generate documentation, and create the thing you give your users. It may also be able to do other things, which is great. Most build systems today do everything but run a flight simulator, but that can’t be far off. ³

Death by Gantt

From a project management point of view, one of the most important things you need to figure out is how you will track the project while you’re in the middle of it.

The most commonly used tool when planning and tracking is the dreaded Gantt chart. I’ve loathed the Gantt chart for a long time but I’ve finally figured out that I hate it because my early exposure to it was using it as a tracking tool. And for tracking the progress of software development, there are few tools worse suited than the Gantt chart. The Gantt displays all tasks as independent and of fixed length. But tasks in the development world tend to be far more malleable, interconnected, and incremental than it’s possible to represent in a Gantt.

Faced with this mismatch, you have two choices: 1) ignore what’s actually happening in the project and approximate it in your rigid schedule or 2) modify the Gantt on a daily basis to reflect the actual state of the project down to the minute. The first will cause developers to ignore the schedule because the schedule seems completely disconnected from reality. The second is madness because the amount of daily churn means the project manager will do nothing else and he will be pestering developers constantly as he does it. These both suck.

For every feature, you should:

have a list of tasks to be completed (this evolves constantly)
know the order they need to be done in (but I find this to be intuitive and not necessary to spell out)
know whether tasks are completed, in progress, or not started
know who is doing them
know estimates for each remaining task

There are a bunch of ways to do this kind of tracking during the life of a project. I’ve most commonly used a spreadsheet or plain text file, but have had some success with agile-oriented PM tools like Rally or VersionOne. The point is that for many, many projects you can track this stuff with little work by relaxing your death grip on the initial schedule. ⁴

You do need to know how much work remains so that you can either adjust end dates or de-scope by dropping or shaving features. You can determine this with: (tasks remaining * estimates) – people time available. If that’s <= 0, you need to take corrective action. I find doing this adjustment on a weekly basis works pretty well and can take less than an hour if you stay on top of it. A burn-down chart is a really nice way to represent this info.

The XP and agile guys will also talk about velocity, which is nothing more than a feedback loop (here we go again) that helps you dial in estimates to reality. That’s cool and useful but in self-aware teams usually tells you something you already know.

Coming up next…the End-Game!

Notes

¹ Cycles in cycles reminds me of the wonderful and possibly apocryphal story: A scientist is giving a lecture about the nature of the universe and after the lecture a woman approaches him and says “This is ridiculous; everyone knows the world is flat and rests on the back of a giant tortoise.” The scientist asks, “What is the turtle standing on?”. The woman replies, “Very clever, but it’s turtles all the way down.”

² Fred Brooks was half right – build to throw two away.

³ Lazyweb: waiting for mvn flight:simulator.

⁴ I am aware of course that all the info I listed is exactly the same info that already exists in a Gantt chart. There are multiple reasons why Gantt charts FAIL anyways. First, they are too precise. By specifying exact start and end points for every possibly task and assigning orderings, dependencies, and people you present a picture of rigid control even though this bears no relation to how the work is actually getting done. As a developer, that always made it feel like bullshit to me. Second, the need for precision means you spend way too much trying to figure out what happened in the past even though that doesn’t matter anymore. Trying to retroactively mine what exact tasks were done, by whom, and to what percentage is missing the point.