Definition of Done

Marcello is considered a brilliant programmer by the team, willing to tackle any task is set upon him, he writes lots of very complicated code in little time. "It will be done in a couple of days", he likes to declare right before diving into his magic. Few days later you can find him typing effortlessly on the keyboard, immersed in his thoughts: "A couple more days, it's going to be very cool".
Some time later, he finally announces triumphantly: "It's done!". The new code is immediately committed to the main line ready to be tried out by designers, eager to cut their teeth with the new shiny piece of technology they were impatiently waiting for.

The new code crash the build.
This story is far too common in our world, but i wouldn’t blame the programmer, often very passionate about his work, so willing to please the hungry designers. The pressure to deliver features is often enormous, bugged and untested features are released, more time is spent fixing past features than developing new ones. And pressure piles up.
I blame the lack of a clear  Definition Of Done, that is able to define without ambiguities when a feature is ready to be deployed to the rest of team and go into production.
The final goal of the Definition of Done is to increase the rate at which working features are produced by the programming team in the long term.
The most common Definition Of Done that I've encountered in the Industry (and often used myself) is something similar to:
It's done when I wrote the code and it works on my machine.
This Definition Of Done is far from being unambiguous: in fact it means different things for each member of the team. To begin with, it's not clear what we mean here by "it works". Have I tried any possible condition in game? Or it works just in my testbed and I'm assuming it works in the game as well? Has someone from QA done a play through? Has he smoke tested the entire game or just a level? Has someone from the design team tried to use the feature? Does it actually do what the designers need? Do all automated tests pass? Is there any automated test at all?
And what about code quality: can code be changed easily if something doesn't work (and it will not work) or if something needs to be modified later (and it will need to be modified)?
Too many open questions without clear and unique answers to make this definition safe for the team. When a feature is "done", the content creators or other engineers can work on it with confidence, knowing that the chance of crashing or not doing what was asked for are minimized. Not zero. Just minimized.
A more useful Definition Of Done is a checklist of activities that needs to capture, more or less closely, the following concepts:
Uniqueness. A feature is "done" at the same stage of development regardless of who has been working on it.
Automation. Criteria can be checked automatically by tools, impact on actual production is minimal.
Quality. Code quality and implementation quality are guaranteed.
Agreeability. The entire team has to agree and stick to the Definition of Done, including designers and management.

The best defense is a good offense

 

My students might be moving their first careful steps in the dangerous lands of programming, but the problems I’m forcing them to face week after week are nothing short than “difficult”, even in code that is seemingly simple and straightforward. This week we were working on a sprite engine in C# (XNA), adding some code to select different sprite animations to play, where an animation is defined as an array of images. The sprite keeps track of the index of the current image being used in the current animation. Pretty straightforward…. crash.

Oh crashes happen, it’s part of the game, and we quickly found out the source of the problem:

Texture2D GetFrame(int index) 
{   
  return frames[index];    
}

An exception, IndexOutOfBound, was being thrown when a new animation is selected: the current index is being maintained by the Sprite class, and it’s not guaranteed that the new animation contains at least the same number of frames of the previous one. GetFrame() can, potentially, receive an index that is out of bound.

Solution: Make sure GetFrame() doesn’t fail if the index is out of bound.

Something like:

Texture2D GetFrame(int index) 
{   
  if (index > frames.Count)     
     index = 0;   
  return frames[index]; 
}

This is a very common approach to the problem that in my opinion is not actually solving it, merely hiding the dirt under carpet of an if guard. Surely, it “fixes” the bug, but at what cost: some added complexity (it’s an if, one more branch and one more unit test to write in an automated tested environment), that hides away the source of the bug; the system can still be in an inconsistent state. We are just trying to keep going in the presence of a blatant fault.

The more stable solution, I explained to the students, is to fix the source of the inconsistency, in this case either having one separate index per animation or, more simply, reset the frame index to zero when a new animation is selected: an animation is guaranteed to always have at least one frame, the code is thus safe, it doesn’t need a check in GetFrame() (one less branch) and it won’t crash… ever.

Unless we have an animation with no frames.

This is precisely the behavior I want: if there’s an animation with no frames, I want the application to crash and tell me if it’s in an inconsistent state that I can not ignore. If it crashes, I have to fix it.

Fail early, fail loud

This simple example shows my students a broader approach to software construction: offensive programming versus defensive programming. Instead of trying to prevent a crash by writing lots of defensive code that checks anything that can possibly go wrong (but obfuscates meaningful and useful code in a see of ifs), I prefer writing less code, but make it so execution stops as early as possible in the presence of inconsistent state, invalid preconditions, broken invariants.

In my experience this approach leads to less and simpler code, easier to modify and with less crashes.

An obvious drawback of offensive programming might be a live system, that is used by clients (content creators?) while it’s being developed: ring a bell? It’s exactly how we work in the Game Industry. Although I’m convinced that it produces less down time in the long run (by producing better and less crash-y code), it’s hard to convince an artist staring at a call-stack that this is a Good Thing™.

I’m again very open to suggestions here, how would you tackle the problem of writing lean and offensive code versus making the content creators experience pleasant in the presence of a (undoubtedly more rare) show stopper?

Some ideas:

• User friendly crash handler
  
• Automated unit, functional and smoke testing before release to minimize crashes
  
• Build staging

I played a lot in the past with unit and functional testing, especially in the form of asset testing to make sure that assets are sanitized before entering the system, with very promising results that led to fairly stable daily release cycles; something very much appreciated by content creators. Still that show stopper is hard to sell…

This post is also available on AltDevBlogADay.

A trip to Berlin

I’ve been teaching game programming at the University of Darmstadt for a couple years now: as they say, teaching something helps you clarifying it in your head. They also say that who can do something does it, who can’t teaches it

Last week I called two “volunteers” as usual to write some code at the main computer in front of the rest of the class, gave them some freedom to choose a solution for a little problem we were having in our XNA game. The solution they came up with was good and, then, they generalized it, writing some more code, and then some more code, and then a little bit more code.

I asked why.

We might need it later

I replied.

So next weekend you want to go to Berlin, spring is finally coming, time to enjoy a bit of sun, it won’t last long here in Germany. You evaluate all the available options, you may take the train (even if they are so expensive here), or go by plane which is not much more expensive than the train and takes less time, or rent a car which would give more freedom but costs a bit more. You discuss it among yourselves for a while, weighing pros and cons against the cost of each option; the train will be around 200 euro per person, same as the a plane ticket, while renting a car would cost around 100 euro per day plus petrol. You finally choose and you buy a car for 5.000 euro. Why? You might need it later. Maybe you will like Berlin so much that you want to travel there every weekend, or you will need the car to go to Amsterdam instead, or even Or maybe you will not do anything like that, and you will spend ten times more now to go to Berlin than if you just hopped on a train…

 

In our daily life we face this kind of tradeoffs and, with material things, we very often rationally choose to spend less now for something we need now, rather than spend more for something we might need later. I’m ignoring for this post the well proved fact that decisions are not always taken rationally (for a good book on the matter look here). In software we usually take the alternative route, we might need it later, we are willing to spend more now for something that more often than not we will not need later. We do it so often that we gave it a name: Over Engineering.

I believe that this peculiar behavior stems from the fact the cost associated to each line of code we write is not directly evident, it’s not money leaving our pockets. It’s also difficult to estimate precisely, which leads to the false perception that generalizing a solution will actually save time. It will not. Each line of code has a cost over time in terms of maintenance and added complexity, that is paid by the team day after day: if it doesn’t solve a problem that we have now, it’s a sunk cost. It requires a huge amount of experience to foresee that “we might need it later” is actually going to be true, and it also requires having gone through the same problem at least few times, which is not always the case in our Industry: we are on the bleeding edge of technology, uncharted territory, where things are complex by nature. Adding more complexity that is strictly needed is not going to help our cause.

Half jokingly with my students, I showed how they could solve the problem with half the number of lines of code, which will probably mean something like half the number of bugs at the end of the project if the idea is applied consistently.

You Aren’t Gonna Need It

No Comment

When I interview a new C++ programmer, my favourite question is not about const-correctness or virtual tables, but: "Do you comment your code?". I still ask about virtual tables, but asking about code comments always springs some interesting discussions and gives me a good insight about the level of care the aspiring programmer puts in crafting his code to not only make it correct and fast, but also to make it easy to work with. I witnessed puzzled faces wondering if that was a trick question (yes it is), and one unlucky candidate pushed his luck to a very bold "which good programmer doesn't comment his code!". Me.

More on Continuous Deployment

This time is the turn of Digg, the popular bookmaring website. Thanks to Jens for finding it out.

http://about.digg.com/blog/continuous-deployment-code-review-and-pre-tested-commits-digg4

No change is going to make it onto our internal site without passing a full battery of unit tests. And nothing is going to go on from our internal site to production without passing an additional set of Selenium tests. Does this mean we're never going to introduce bugs to our live site? Of course not - but we're going to keep the number of bugs to hit the live site to a minimum, and we've made it easy and fast to get bug fixes live as well.

“A New Version Of Google Chrome Now Due Every Six Weeks”

  http://techcrunch.com/2010/07/22/google-chrome-versions/

With their Chrome web browser, Google has always been obsessed with speed. And now they’re speeding up another aspect of it: how often stable builds are released. The goal now is to release a new stable version of the browser every six weeks […]

This is not directly related to game development, but nevertheless very very interesting: to reliably release stable (not beta or dev) build to millions of users every six weeks, the Chrome team needs water tight automated testing, very good and consistent code quality and extremely honed agile processes: also the cost of making a change has to be small, while debugging relatively predictable. There is no room for long check-in queues, deep branch hierarchies, check-in rights, locked branches and long crunching months to meet a deadline. I can imagine that Continuous Integration and Release Branches are keys to meet this ambitious goal.

On a smaller but still interesting scale, Wordpress is on the same page:
http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/

Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at a launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.

That’s where the Game Industry has lots to learn: the concept of being always in a state to be able to produce a stable build while keeping development up and running in a sustainable environment will prove to be the next Big Thing in Game Development to give some companies the competitive edge to release good games in time. A dream? What’s your opinion?

Festering code bases and budding code bases

A truly excellent read:
http://michaelfeathers.typepad.com/michael_feathers_blog/2010/06/festering-code-bases-and-budding-code-bases.html

Especially the concept that at the time of change, just adding code to an existing method, and not refactoring to a better structure, is perceived as more cost effective, but it’s just accumulate technical debts, that over time cost money in the form of interests. Same concept can be applied to write unit tested code: it seems to cost more at the time of change, but costs less in the long term, by flattening the curve of cost-of-change.

AAA Automated Testing

My slides on my automated testing experience in Crytek presented at GDC Europe 2009.
Warning: it contains explicit material.

AAA Automated Testing