The best defense is a good offense

 

My students might be moving their first careful steps in the dangerous lands of programming, but the problems I’m forcing them to face week after week are nothing short than “difficult”, even in code that is seemingly simple and straightforward. This week we were working on a sprite engine in C# (XNA), adding some code to select different sprite animations to play, where an animation is defined as an array of images. The sprite keeps track of the index of the current image being used in the current animation. Pretty straightforward…. crash.

Oh crashes happen, it’s part of the game, and we quickly found out the source of the problem:

Texture2D GetFrame(int index) 
{   
  return frames[index];    
}

An exception, IndexOutOfBound, was being thrown when a new animation is selected: the current index is being maintained by the Sprite class, and it’s not guaranteed that the new animation contains at least the same number of frames of the previous one. GetFrame() can, potentially, receive an index that is out of bound.

Solution: Make sure GetFrame() doesn’t fail if the index is out of bound.

Something like:

Texture2D GetFrame(int index) 
{   
  if (index > frames.Count)     
     index = 0;   
  return frames[index]; 
}

This is a very common approach to the problem that in my opinion is not actually solving it, merely hiding the dirt under carpet of an if guard. Surely, it “fixes” the bug, but at what cost: some added complexity (it’s an if, one more branch and one more unit test to write in an automated tested environment), that hides away the source of the bug; the system can still be in an inconsistent state. We are just trying to keep going in the presence of a blatant fault.

The more stable solution, I explained to the students, is to fix the source of the inconsistency, in this case either having one separate index per animation or, more simply, reset the frame index to zero when a new animation is selected: an animation is guaranteed to always have at least one frame, the code is thus safe, it doesn’t need a check in GetFrame() (one less branch) and it won’t crash… ever.

Unless we have an animation with no frames.

This is precisely the behavior I want: if there’s an animation with no frames, I want the application to crash and tell me if it’s in an inconsistent state that I can not ignore. If it crashes, I have to fix it.

Fail early, fail loud

This simple example shows my students a broader approach to software construction: offensive programming versus defensive programming. Instead of trying to prevent a crash by writing lots of defensive code that checks anything that can possibly go wrong (but obfuscates meaningful and useful code in a see of ifs), I prefer writing less code, but make it so execution stops as early as possible in the presence of inconsistent state, invalid preconditions, broken invariants.

In my experience this approach leads to less and simpler code, easier to modify and with less crashes.

An obvious drawback of offensive programming might be a live system, that is used by clients (content creators?) while it’s being developed: ring a bell? It’s exactly how we work in the Game Industry. Although I’m convinced that it produces less down time in the long run (by producing better and less crash-y code), it’s hard to convince an artist staring at a call-stack that this is a Good Thing™.

I’m again very open to suggestions here, how would you tackle the problem of writing lean and offensive code versus making the content creators experience pleasant in the presence of a (undoubtedly more rare) show stopper?

Some ideas:

• User friendly crash handler
  
• Automated unit, functional and smoke testing before release to minimize crashes
  
• Build staging

I played a lot in the past with unit and functional testing, especially in the form of asset testing to make sure that assets are sanitized before entering the system, with very promising results that led to fairly stable daily release cycles; something very much appreciated by content creators. Still that show stopper is hard to sell…

This post is also available on AltDevBlogADay.

A trip to Berlin

I’ve been teaching game programming at the University of Darmstadt for a couple years now: as they say, teaching something helps you clarifying it in your head. They also say that who can do something does it, who can’t teaches it

Last week I called two “volunteers” as usual to write some code at the main computer in front of the rest of the class, gave them some freedom to choose a solution for a little problem we were having in our XNA game. The solution they came up with was good and, then, they generalized it, writing some more code, and then some more code, and then a little bit more code.

I asked why.

We might need it later

I replied.

So next weekend you want to go to Berlin, spring is finally coming, time to enjoy a bit of sun, it won’t last long here in Germany. You evaluate all the available options, you may take the train (even if they are so expensive here), or go by plane which is not much more expensive than the train and takes less time, or rent a car which would give more freedom but costs a bit more. You discuss it among yourselves for a while, weighing pros and cons against the cost of each option; the train will be around 200 euro per person, same as the a plane ticket, while renting a car would cost around 100 euro per day plus petrol. You finally choose and you buy a car for 5.000 euro. Why? You might need it later. Maybe you will like Berlin so much that you want to travel there every weekend, or you will need the car to go to Amsterdam instead, or even Or maybe you will not do anything like that, and you will spend ten times more now to go to Berlin than if you just hopped on a train…

 

In our daily life we face this kind of tradeoffs and, with material things, we very often rationally choose to spend less now for something we need now, rather than spend more for something we might need later. I’m ignoring for this post the well proved fact that decisions are not always taken rationally (for a good book on the matter look here). In software we usually take the alternative route, we might need it later, we are willing to spend more now for something that more often than not we will not need later. We do it so often that we gave it a name: Over Engineering.

I believe that this peculiar behavior stems from the fact the cost associated to each line of code we write is not directly evident, it’s not money leaving our pockets. It’s also difficult to estimate precisely, which leads to the false perception that generalizing a solution will actually save time. It will not. Each line of code has a cost over time in terms of maintenance and added complexity, that is paid by the team day after day: if it doesn’t solve a problem that we have now, it’s a sunk cost. It requires a huge amount of experience to foresee that “we might need it later” is actually going to be true, and it also requires having gone through the same problem at least few times, which is not always the case in our Industry: we are on the bleeding edge of technology, uncharted territory, where things are complex by nature. Adding more complexity that is strictly needed is not going to help our cause.

Half jokingly with my students, I showed how they could solve the problem with half the number of lines of code, which will probably mean something like half the number of bugs at the end of the project if the idea is applied consistently.

You Aren’t Gonna Need It

No Comment

When I interview a new C++ programmer, my favourite question is not about const-correctness or virtual tables, but: "Do you comment your code?". I still ask about virtual tables, but asking about code comments always springs some interesting discussions and gives me a good insight about the level of care the aspiring programmer puts in crafting his code to not only make it correct and fast, but also to make it easy to work with. I witnessed puzzled faces wondering if that was a trick question (yes it is), and one unlucky candidate pushed his luck to a very bold "which good programmer doesn't comment his code!". Me.

No comment

while (IsStarted())
{
    // Wait until there is work to do.
    Wait();

Oh well, I guess so…

Crunch… crunch… crunch... #2

 

Old and very well known article, but it’s worth a refresh cause it seems to be overlooked:
http://archives.igda.org/articles/erobinson_crunch.php

In most times, places, and industries over the past century, managers who worked their employees this way would have been tagged as incompetent — not just because of the threat they pose to good worker relations, but also because of the risk their mismanagement poses to the company's productivity and assets. A hundred years of industrial research has proven beyond question that exhausted workers create errors that blow schedules, destroy equipment, create cost overruns, erode product quality, and threaten the bottom line. They are a danger to their projects, their managers, their employers, each other, and themselves.

Crunch… crunch… crunch…

http://www.estherderby.com/weblog/archive/2006_03_01_archive.html#114167242943470460

Extended periods of overtime are more likely to SLOW DOWN work than speed up work.

Rather than demand overtime,

Look at the universe of work for the department, not just individual work loads

Decide what to do and *what not to do* based on the mission of the group

Prioritize the work the group needs to do

Determine the level of quality required for work – what “done” means and what “good enough” means

Collect data on how people spend their time at work – not to beat people over the head, but to look at patterns that may be lowering productivity
(Typical time-robbing culprits are: Multi-tasking, fragmentation, frequent interruptions, meaningless meetings, non-value adding corporate activities.)

More on Continuous Deployment

This time is the turn of Digg, the popular bookmaring website. Thanks to Jens for finding it out.

http://about.digg.com/blog/continuous-deployment-code-review-and-pre-tested-commits-digg4

No change is going to make it onto our internal site without passing a full battery of unit tests. And nothing is going to go on from our internal site to production without passing an additional set of Selenium tests. Does this mean we're never going to introduce bugs to our live site? Of course not - but we're going to keep the number of bugs to hit the live site to a minimum, and we've made it easy and fast to get bug fixes live as well.

“A New Version Of Google Chrome Now Due Every Six Weeks”

  http://techcrunch.com/2010/07/22/google-chrome-versions/

With their Chrome web browser, Google has always been obsessed with speed. And now they’re speeding up another aspect of it: how often stable builds are released. The goal now is to release a new stable version of the browser every six weeks […]

This is not directly related to game development, but nevertheless very very interesting: to reliably release stable (not beta or dev) build to millions of users every six weeks, the Chrome team needs water tight automated testing, very good and consistent code quality and extremely honed agile processes: also the cost of making a change has to be small, while debugging relatively predictable. There is no room for long check-in queues, deep branch hierarchies, check-in rights, locked branches and long crunching months to meet a deadline. I can imagine that Continuous Integration and Release Branches are keys to meet this ambitious goal.

On a smaller but still interesting scale, Wordpress is on the same page:
http://toni.org/2010/05/19/in-praise-of-continuous-deployment-the-wordpress-com-story/

Launching products is one of the hardest things companies do. Most companies pour months of work into making sure everything goes right at a launch – the features are right, the marketing is ready, the press is primed, the product is solid, etc. But a new breed of companies are doing things very differently. Instead of optimizing product launches to go as perfectly as possible, they optimize to have them go as quickly as possible.

That’s where the Game Industry has lots to learn: the concept of being always in a state to be able to produce a stable build while keeping development up and running in a sustainable environment will prove to be the next Big Thing in Game Development to give some companies the competitive edge to release good games in time. A dream? What’s your opinion?

Festering code bases and budding code bases

A truly excellent read:
http://michaelfeathers.typepad.com/michael_feathers_blog/2010/06/festering-code-bases-and-budding-code-bases.html

Especially the concept that at the time of change, just adding code to an existing method, and not refactoring to a better structure, is perceived as more cost effective, but it’s just accumulate technical debts, that over time cost money in the form of interests. Same concept can be applied to write unit tested code: it seems to cost more at the time of change, but costs less in the long term, by flattening the curve of cost-of-change.