The standard improvement is to use fixtures, a file with some example data, that your tests can load by giving the name of a fixture. Here’s one of the fixtures for a gamer:

alice: # alice is heavily involved
  id: 1
  mail: alice@example.com
  handle: alice
  password: red queen
  teaser: Follow the white rabbit.
  profile: Not so plain, after all.
  homepage: http://www.example.com
  jabber: alice@example.org
  location_id: 3
  last_login: 2006-10-30 11:56:08
  created_at: 2006-10-30 11:56:08
  location_at: 2006-10-30 11:56:08
  tag_text: dnd, wod
  mail_on_message: true
  mail_on_nearby_discussion: true

This is straightforward, but location_id is the id of another fixture in another file. As you can imagine it’s pretty easy to get them out-of-sync and break some tests. I end up leaving little comments in the YAML to remind myself of the cross-reference.

It’s awfully tempting to reuse fixtures from test to test. Maybe Alice was written to test logging in with a username and password and next I’m writing the test of logging in with an email address and a password. I could copy and paste Alice to another fixture to reuse in that test, but as a programmer I have a pathological aversion to copy and paste. Alice is going to get reused (most of those fields are optional and were used by very different tests), and after this happens a few times it’s hard to change a fixture without breaking an unrelated test.

I’ve gotten a lot of use out of fixtures as I’ve developed the habit of testing all my code, but reuse has made my tests somewhat brittle. After a bit of research, I’ve moved over to using factories.

Here’s the factory for a gamer:

 
Factory.sequence :handle { |n| "Gamer_#{n}" }
 
Factory.define :gamer do |g|
  g.handle { Factory.next :handle }
  g.mail { |m| "#{m.handle}@example.com" }
  g.password 'secret'
end

Only the essential, required fields are listed. Using this simple template, the tests themselves specify the values of the fields they’ll be testing.

 
def test_login_with_valid_username_and_password
  gamer = Gamer.create :handle => 'Alice', :password => 'red queen'
  ...
end
def test_login_with_valid_email_and_password
  gamer = Gamer.create :mail => 'alice@example.com', :password => 'red queen'
  ...
end

Ideally, changing a fixture wouldn’t break any tests at all. And factory_girl (what I’m using for factories in Rails) makes associations easy:

 
Factory.define :post do |p|
  p.association :discussion, :factory => :discussion
  p.association :poster, :factory => :gamer
  p.created_at Time.now.utc
  p.body "Post Body"
end

Now my tests include the specific data they care about, making them easier to understand and improve. And they don’t interrelate, so they’re much more reliable. I do still use fixtures, but now they’re exclusively for tests that need to deal with real-world examples.

Really, even on personal projects. I write tests because I’m really good at writing buggy software. I’d bet I’m better than you.

I’ve written those bugs that waste hours of time until I found the one missing (or added) bit of punctuation. I’ve written bugs that first-year programming students have caught. I’ve written so many of those really obvious bugs that my boss notices with the first glance that you’d be amazed I’ve never been fired. Bugs that confused my users into thinking they made mistakes. Bugs that ate data. Bugs that corrupt data silently. And lots of those little changes that broke that totally unrelated thing over there that, wait, yeah, it is actually related in this one tiny way. Bugs that would be really obvious if I’d remembered that one bug in IE6, if I’d remembered the API, if I’d remembered what my own code from last week looked like.

I write tests because I’m imperfect and I hate breaking things. I take code seriously and I’m happy when I write the best code I can. When I’m slipping in errors, when I’m wasting the time of myself and my users, and when I’m just plain afraid that I might be introducing bugs, I’m not writing good code. I don’t keep anything on my nightstand except my glasses, because I know I’m just going to knock it all off in the middle of the night when I’m groping for my glasses. I don’t want to set myself up to fail.

Testing isn’t everything, and I’ve talked about its limits. But it’s saved my bacon often enough that I feel like I’m taking needless risks when I don’t write tests.

Yeah, OK, fun little one-off scripts don’t have tests, but that script is also about the largest I’ll let code get without test. Read the comments, someone found that I left a bug lurking to shoot myself in the foot when I tried to expand or reuse the code.

I don’t want to write buggy software, but I know I will, so I write tests and then I build things that are far less buggy. I don’t test my personal projects out of a sense of obligation, I test them so I can relax and have fun building cool things.

That’s why my advice is always to start small with tests. Test what you can easily, you don’t have to write exhaustive tests that cover all possibilities on all your code. Write a happy-path test for when a function or method works right, and cover one or two ways things can go wrong or be misused. A few weeks or months from now when you break it in some silly way or don’t remember how it’s supposed to be called, the time you invested in tests will pay dividends. When some you narrow down a bug, write a test that exercises it, fix the bug, and smile as your test passes: that bug isn’t going to slip back into your code again. You’ll write better code without the stress. Trust me, I know.

I wanted to write to say you’re not refactoring. The term refactoring was created in the context of extreme programming [er, I should have said "popularized"] and it was defined as reworking code that’s under test. Without the tests, you’re just rewriting, working without a net. That’s why your code ends up buggier.

I don’t write to be pedantic. (Well, OK, maybe a little. I am a programmer, after all.) I write because I also used to wonder what all the fuss was about ‘refactoring’ when it really just seemed like a dangerous bunch of nonsense that left my code with new bugs. I didn’t do any automated testing at the time and didn’t realize how valuable it can be.

I still haven’t drunk the XP kool-aid, but having a good test suite means I don’t code in fear that I’ve broken things, or feel awful when a bug slips into production because I didn’t remember to hand-test some weird situation. I’ll even write tests on the personal projects I’m just hacking away privately at — yeah, I probably will just do a little unit testing instead of a really comprehensive, robust system, but even a bit helps.

He responded that a game like Unangband is especially hard to test because it has so many moving parts and much of the gameplay is generated. It does have a very high-level test, an automated player. I wrote back:

I don’t really think anything is too complex for test cases. You have a ‘pick up’ command that, if the player is standing on an object and has inventory space, removes it from the world and puts it in their inventory. There’s a fair amount of data structure you have to mock (which is not the easiest thing in the world in C), but you can know the basic command works even because you test all the really weird corner cases about when the player is polymorphed into a creature without hands or something. I’d bet as simple as the command usually is, you’ve broken ‘pick up’ at least a few times. It’s a safety net you build one strand at a time, like a spider web. You may still be able to fall through (especially at first), but sometimes you’ll catch a strand and be glad you built it for yourself.

The other part of complexity is that thinking about testability changes how you structure your programs. You force yourself to decouple different parts of code because you want to be able to test them in isolation from each other — that this improves your overall design is a nice side-effect. That you can spend less time worrying about or kicking yourself for breaking something ‘so obvious!’ is the real one.

You may enjoy the book Refactoring: Improving the Design of Existing Code, it’s the classic in the space. Though my money’s on the later book Working Effectively with Legacy Code, which is a couple hundred pages on how to go from cursing your predecessors to a reliable codebase. Saved my bacon at a previous job. I know Unangband is your hobby rather than your job, but I suggest them because they’ve helped keep my hobby code more fun than frustrating.

Part of what keeps developers from testing more is friction. You have to pick a test suite, read the docs, learn the API, blow an hour or few chasing a bug because you misunderstood something, figure out the philosophy of it, rewrite your tests so they’re clear and paradigmatic…

The friction seems to be a lot higher in C/C++ than in dynamic languages. There are good technical reasons for it, but I think at least part of it is cultural. I’ve noticed a less testing in the Python community than Ruby, and I think that’s just because Rails was big on testing and has driven so much of the interest in Ruby. The community assumption is that software includes tests.

Well, you run test it once or twice with a file or two you have laying around and don’t get an exception and there’s the file in your S3 file browser. Now do you know it works?

There’s that nagging feeling of doubt that your code might be imperfect that you only get after you’ve been programming long enough to really screw something up. So you write some unit tests to make sure you’re calling the API correctly with neat mocking and all that jazz. You’re an experienced (read: paranoid) developer, so you do more than “happy path” testing, you make sure it raises errors when given keys that contain invalid characters or are too long. Now do you know it works?

Well, no, you haven’t seen it in production. So you test it there and you learn that every few thousand HTTP requests, S3 throws a temporary error and it works fine on retry. You change your code to notice this and try again a few times and only then raise an exception. You’re smart, you encapsulate this somehow so that all your S3 requests use it and you Don’t Repeat Yourself. Now do you know it works?

There’s no more of those random errors in production, but you don’t know it works. You know that you PUT a file to S3 without errors, but you don’t know that you succeeded at the original goal, which was to store a file. The HTTP PUT is the means, not the end.

So you turn around and do an HTTP HEAD against that key you just PUT to make sure the Content-Length matches the size of the file you just uploaded. Heck, if you’re really paranoid, you turn around and GET the file from S3 and hash it to make sure they both have the same SHA1. You’re ensuring that S3 serves up the file you thought you stored. (Maybe you even do this from EC2 to avoid the time and charge of downloading the file, but I’m not going to go into it.) Now do you know it works?

No. Here your troubles begin.

You have a race condition. If another process does a PUT between your PUT and your HEAD/GET, your function will wrongly think Amazon failed to store the file. You can check that the HEAD’s Last-Modified matches the Date on the PUT so you know the file was replaced since you PUT it there, but at that point all your code can do is PUT it again (and boy will it suck when two copies of this program are running and they both want to have the last word) or your code can throw its hands up and pretend things went fine. Now do you know it works?

No, you don’t. You can’t know. Computers promise to be deterministic and reliable, but anything with a little complexity turns into a computer system to sneak in side effects and temporal coupling and other beasties and that’s even before you have to deal with hardware failures or someone turning on the microwave and disrupting your wifi connection…

Finally, You Come to the Point

It’s a trade-off. Even in computers, there is no such thing as perfectly reliable code, there’s only how much resources you’re willing to pay for the next improvements. It’s easy to be oblivious to this. I can’t even count the number of times said some variation of “OK, now I know it works” before I learned that it will always break. Always.

Failure is inevitable. If you want to build anything large and to last, you have to incorporate it into your design and build organic systems that monitor, recover, repair, and, above all, fail gracefully when they themselves inevitably fail.

P.S to folks coding in Ruby: don’t use AWS::S3, it only checks for valid keys on HTTP PUT and doesn’t retry transient errors. RightAWS is an improvement.

Updated to Rails 2.0.1

• rm’d lib/slash_urls.rb: Rails switched from ; to / to separate actions
• rm’d lib/resource_requirements.rb: now included in ActiveResources
• rm’d app/helpers/tags_helper.rb and gamers_helper.rb that had old controller names and were unused anyways
• app/controllers/gamers_controller.rb: documented and fixed raw post variable extraction
• updated post_path linkers in many views, controllers, and tests
• renamed discussion_anchor_post_path to anchor_discussion_post_path to match the new ordering for nested resource urls
• fix tag sorting on tag index
• move tags from /tags/Board+Games to /tags/Board_Games, as Rails fixed the bug that encoded ‘ ‘ as ‘+’ in non-get variable parts of the URL

The Details

The first two files I deleted were my patches to URL generation that Rails now includes. The next two helper files I never used, but had had old (singular noun) names.

The bit about documentation: the way to get at raw post variables changed, so I tweaked code and left myself a reminder of why I’m doing it (I had to look at the changelog, which means it wasn’t self-evident).

Nested resources (I have nested map.resources :discussions { |d| d.resources :posts }) changed from discussion_edit_post_url to edit_discussion_post_url, so there was a bunch of places to change that. The next change about anchor is me following this convention with my own convenience method.

Tag sorting on the index page was a bug that Snarky pointed out to me. When I made some performance tweaks to that page I forgot to keep sorting tags as they came out of the database. This arguably should’ve been a revision of its own as it’s unrelated to the upgrade, but it was an easy one-line bugfix so I let it in.

Last, I renamed the tag pages. The tag model’s to_param just returns self.name and Rails would turn “Board Games” into “Board+Games”. It’s common but technically incorrect, as + only means space in encoded GET/POST variables. The proper encoding of “Board%20Games” was really noisy, so I took a page from Wikipedia’s pagebook and use underscores instead, like “Board_Games”. Was one line in the to_param and one line in the controller’s load_tag before_filter, a dozen lines of tweaking old tests, and seven lines of new tests. Eeeeeasy.

The Verdict

Tests, tests, tests. If I didn’t have a solid test suite, I’d be noticing bugs two months from now in the least-frequently-exercised bits of code. Not only would I not have remembered to test some of the functionality, I wouldn’t have remembered the corner cases — or maybe I’d just have gotten bored of clicking through pages over and over and ignored the corner cases.

None of these were hard fixes and none were frustrating. I’d probably have finished even sooner if I hadn’t also been poking lolcats and chatting with friends. And now I have a big list of shiny new Rails 2.0 features I can put to work in NearbyGamers.

I’m now a big fan of Mocha like I am of Rails. Ruby has a way of allowing coders to write magically polished libraries.

I’m writing some code (“Fetcher”) to talk to a POP3 server, fetch mail, and pass it off to another process. One of the tests deals with what happens when the POP3 server is down or otherwise unreachable.

 
def test_setup_server_down
  pop3 = Mock.new
  pop3.expect(:new, [MAIL_SERVER, MAIL_POP3_PORT]){ raise Timeout::Error.new("execution expired") }
  f = Fetcher.new(pop3)
  # further assertions that f acts correctly
end

This says that the Mock object expects a call to the new method with the given arguments, and when the call happens it runs the block. The block could return anything, but in this case it raises the same error as Net::Pop3 does when it can’t contact the server. After that the test can go on to make whatever assertions it needs to verify that the exception was handled properly.

The Mock object has a list of calls it expects to see and keeps a list of how it’s been called (yes, this could just be one list with an index but I thought it was mentally simpler this way). The test sets up what calls it should expect to see with what arguments (or blank for any) and block to run (or blank for none). When a method is called on the mock object, method_missing logs the call and executes the given block (raising a fuss if the call didn’t match what it expected).

 
class Mock
  attr_reader :calls, :called
 
  # the stub arg makes it just record all calls
  def initialize
    @calls = []
    @called = []
  end
 
  # Pass nil for args to ignore the actual args in the call.
  # Proc is optional; default is empty proc returning nil.
  def expect(method, *args, &proc)
    @calls << {:method => method, :args => args.first, :proc => (proc or Proc.new{})}
  end
 
  def method_missing(method, *args)
    @called << {:method => method, :args => args}
 
    expect = @calls.shift
    raise "Unexpected mock call #{method.to_s}(#{args.join(', ')})" if expect.nil?
    raise "Wrong mock call #{method.to_s}(#{args.join(', ')}); expected #{expect[:method]}(#{expect[:args].join(', ')})" if method != expect[:method] or (expect[:args] != nil and args != expect[:args])
    expect[:proc].call(*args)
  end
end

It’s a straightforward little object, and I also added some code to raise a fuss if expected calls weren’t made. This does have the downside that any tests defining their own teardown need to call super.

 
class Mock
  def fail_if_not_empty
    # Empty the call stack so that this obj doesn't throw errors for
    # every later test between now and this object getting gc'd
    calls, @calls = @calls, []
    raise "Mock calls uncalled: n" + calls.collect { |call| "#{call[:method]}(#{call[:args]} { #{call[:proc] })" }.join(" ") unless calls.empty?
  end
end
 
class Test::Unit::TestCase
  def teardown
    finish_mocks
  end
 
  def finish_mocks
    ObjectSpace.each_object(Mock) do |m|
      m.fail_if_not_empty
    end
  end
end

This has been a handy piece of code to test the code I’ve written in the last two weeks, but it’s not good enough. I have to use a technique called dependency injection to test Fetcher.new, where the outside code passes it a POP3 object instead of its initialize just using Net::POP3. Useful for testing, but my code is badly repetitive when all the instantiation calls have to do this exact same setup. (As an aside, Jacob Proffitt recently started an interesting conversation on dependency injection, took criticism, and responded. Good reading.)

I was pondering how to extend the Mock object to let me mock class methods (eg a call to Net::POP3.new) when I realized I’d gone as far as I should down the do-it-yourself road. I’d heard of Mocha and it took me all of ten minutes to think to look at its cheat sheet where that’s the first example.

After spending two hours or so writing and tweaking this code, the best thing for it is to be thrown away. I’ve learned about mocking by doing it and I’m better-prepared to understand someone else’s larger and better library.

Now it’s time to treat it as real code, which means getting it under test. Tests are especially useful for ensuring tricky edge cases like malformed input don’t quietly break during other changes. When I started writing unit tests for the sorter I wanted fixtures similar to the Rails fixtures so I could write:

 
require File.dirname(__FILE__) + '/../test_helper'
require 'message'
 
class MessageTest < Test::Unit::TestCase
  fixtures :message
 
  def test_initialization
    f = Message.new message(:good)
    # assert the object was parsed correctly, etc.
  end
end

The basic usage is fixtures :type to load the data and create the named function (in this case, message) for fetching data. First, I wrote some YAML fixtures for my data and put them in test/fixtures/message.yaml. You can see the :good key in use here:

good: |
  [the body of a good message here]
missing_header: |
  [more text]
bad_checksum: |
  [more text]

Because I’m parsing textfiles my fixtures look pretty simple, but the YAML library for Ruby makes serialization of any very easy.

So here’s the code I put in test/test_helper.rb to set this up. I really like the [list].flatten.each idiom so the call from the unit test class can cleanly pass one or more types, eg: fixtures :message and fixtures :message, :user.

 
class Test::Unit::TestCase
  @@fixtures = {}
  def self.fixtures list
    [list].flatten.each do |fixture|
      self.class_eval do
        # add a method name for this fixture type
        define_method(fixture) do |item|
          # load and cache the YAML
          @@fixtures[fixture] ||= YAML::load_file("test/fixtures/#{fixture.to_s}.yaml")
          @@fixtures[fixture][item.to_s]
        end
      end
    end
  end
end

(Note to Rails coders: I deliberately left out the pluralization, I like having the File class in file.rb with the fixtures in text/fixtures/file.rb and the database table named file and so on. I don’t mind Rails’ convention of pluralizing to “files” in the latter two examples, but I’d rather have a single identifier.)

Last night at the Rudolph Hering Society meeting Ian Bicking and Atul Varma explained mocking to me. I wrote a (very) little Ruby this morning to do mocking and I’ll post that in a day or two when I’m sure I didn’t do it too badly.

It sounds like you’ve given up on software development. You’re drawing a line in the sand at the outside edge of a program to say, “Here, this far and no further, here is where I can objectively judge software. This is my one firm place from which I’ll move the world.”

You’re not wrong, it’s a good place to make a stand for quality. But you’re not right when the nicest thing you can say about a program is, “Well… it works.”

I don’t want to be poisoned when I go to a restaurant, yes, that’s the minimum bar. But I want fine food, not just fast food. I want to be delighted by my meals, not merely satiated. The software market suffers from a staggering demand and the barely competent who have stumbled up to meet it. If the marketplace could decide quality, the Daily WTF would not exist, but it’s a market for lemons.

I code because I love writing good code. It keeps me up at night — hell, it wakes me up at night. I don’t code so I can tell myself I’ve adequately performed my duties, I code because I love writing the best code I can and then pushing the definition of what my best is.

Maybe I’m ranting away at a false dichotomy. A certification based on testing is a great idea and I’d hit the books to earn it. I just don’t think it’s enough.