As a web developer, I have a half-dozen for testing reasons, but one that I use for all of my personal browsing and most of my work. Very infrequently, I’ve noticed that it sometimes would have errors on web pages that other Firefox profiles didn’t. After some thinking, I realized that I’ve been copying the same Firefox profile nearly ten years, copying it between computers as I upgrade. And all the while I’ve been tinkering with it, poking around in its config files, testing extensions, changing options in about:config.

I wanted a fairly mindless activity today, so I recreated my profile from scratch, re-installing all my extensions and configuring everything just the way I like it. It took a few hours, and my notes ended up surprisingly long.

The few times I’ve thought about switching to Chrome I’ve looked for one or two of these add-ons and seen that there’s no equivalent. Now that I have these notes I can check comprehensively the next time I get the urge.

The notes will mostly be of interest to web developers. My browser is pretty heavily customized with tools, privacy protections, and productivity tweaks (mostly to make it feel like vim and allow me to pull things offline easily). It’s an intimidating list to look at, but it’s ten years of small changes every few weeks all at once.

Add-Ons

• AdBlock Plus http://adblockplus.org/en/
  • add EasyList (button in tab on startup)
  • ‘Options’ menu, uncheck ‘Show tabs on Flash and Java’
• Better Privacy
  • select any LSOs you want to keep
• Cookie Monster
  • uncheck “Automatically Reload…”
• Download Statusbar
  • General
    • check ‘Download speed’
    • uncheck ‘Keep a download history
    • check ‘Clear finished downloads when the browser closes’
    • ‘Automatically clear these filetypes’: *
    • after 90 seconds
  • Appearance – choose ‘Custom Style’
    • File Name Size: 9
    • Height: 13
    • check ‘Main Downloads Button’
    • check ‘Clear Finished Button’
    • check ‘Enable Speed Colors’
• FacebookBlocker
• Flashblock
  • check ‘Block Silverlight as well’
  • add any whitelist sites
• Greasemonkey
• LeechBlock
  • block tvtropes.com
  • General – check all
• Pentadactyl

  ~/.pentadactylrc

  "1.0b4.3 (created: 2011/01/05 17:55:12)
  
  loadplugins '\.(js|penta)$'
  set guioptions=BrsC
  set runtimepath=/home/harkins/.pentadactyl
  set urlseparator=,,s
  
  " vim: set ft=pentadactyl:
  

• NoSquint
  • Zooming tab, uncheck ‘Show current zoom levels’
  • Set ‘Default full page zoom level’ to 100%
  • Exceptions tab, add .github.com, .tumblr.com, .blogspot.com, .posterous.com
• Pixlr Grabber
  • uncheck ‘To edit images’ and ‘To edit backgrounds’
  • select ‘Always save to desktop’
• Readability https://www.readability.com/addons
• RefControl
  • click ‘Edit’ button, choose ‘Block’ and check ’3rd Party requests only’
  • outside of settings, right click button at bottom of window, remove icon
• Sauce Launcher https://addons.mozilla.org/en-us/firefox/addon/sauce-launcher/
• ScrapBook
  • Show up to: 10
• SQLite Manager
  • select ‘in a new tab’
  • uncheck ‘Always confirm’
  • uncheck everything in Prompts tab
• Tamper Data
• Tree Style Tab
  • Appearance
    • Skin ‘Sidebar’
    • Tree Twisties: choose ‘None’
  • Menu
    • uncheck all but ‘Reload this Tree’ and ‘Close this Tree’
  • New Tabs
    • uncheck ‘Always ask’
    • select ‘Open in new tabs’
  • Tree
    • uncheck ‘When a tab gets focus’
    • check ‘Double-click on a tab’
  • Advanced
    • uncheck ‘Enable Animation Effects’
    • uncheck ‘Show tree contents’
• UnPlug
  • Integration – uncheck all but tools menu
  • Downloads – ‘Preferred download method’: Save as
• User Agent Switcher
• Web Developer
  • General
    • Check to hide the context menu, confirmation, and informational
  • Validation
    • select CSS 3

Incompatible

These aren’t yet compatible with Firefox 5, so they didn’t get into my notes.

• Page Speed
• Resurrect Pages
• YSlow

GreaseMonkey Scripts

Sorry for the lack of titles, but this was annoying enough to compile already.

51281 62706 7705 23555 53547 60314 38516 31804 56690 100746 104489 38985 45032 75263 77801 94112 30646 23175 28374 29279 29682 29681 29829 29828 95298 29666 49366 74340 76300 45032

Toolbars

• right click on the toolbar, uncheck Web Development toolbar
• right click again, choose ‘Customize’, choose ‘Use Small Icons’ and remove:
  • read it later
  • search box
  • zoom buttons
  • home button
  • bookmarks button
  • FireBug button
  • drag GreaseMonkey button to status bar

Cookies

click on CookieMonster at bottom of window, click View Cookies, show exceptions.
‘Allow’ sites you regularly visit
‘Block’ google properties

Bookmarks menu -> Show all Bookmarks

‘Import and Backup’ -> Back up to json in old profile, Import ‘Choose File’ in new
Same with ScrapBook – open sidebar, Tools, Import / Export, select all…

Edit -> Preferences

• General
  • ‘When Firefox starts’: Show my windows and tabs
  • ‘Home Page’: about:blank
• Tabs
  • check everything but ‘When I open a link in a new tab’
• Applications
  • search for VLC, Windows, and QuickTime, change items to ‘Always Ask’
• Privacy
  • check ‘Tell web sites I do not want to be tracked’
  • select ‘Firefox will’: Use custom settings for history
  • select ‘Keep until’: I close Firefox
  • select ‘When using the location bar, suggest’: History
• Security
  • uncheck ‘Remember passwords for sites’
• Advanced -> Update
  • uncheck ‘Search Engines’

about:config

• browser.autofocus false
• browser.dom.window.dom.enabled true (new)
• browser.history_expire_days 30
• browser.tabs.animate false
• browser.tabs.closeButtons 2
• browser.tabs.loadBookmarksInBackground true
• browser.tabs.loadDivertedInBackground true
• browser.tabs.selectOwnerOnClose false
• browser.zoom.full false
• browser.jsannoyances.disabled true (new)
• downbar.display.clearButton false
• downbar.display.mainButton false
• downbar.display.percent true
• downbar.function.useAnimation false
• network.http.pipelining true
• network.http.pipelining.maxrequests 8
• network.prefetch-next false

~/.mozilla/firefox/[profile]/chrome

userChrome.css

 
    @namespace url("http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul");
 
    /* don't bold active tab title */
    /* no longer works, argh */
    tab[selected="true"] { font-weight: normal !important; }
 
    /* stop TreeStyleTab's Sidebar theme from changing its color when in/active */
    .tabbrowser-strip[treestyletab-style="sidebar"][treestyletab-mode="vertical"] {
      background-color: #ccc !important;
    }

userContent.css

 
    /* indicate nofollow links with a dotted blue underline */
    a[rel~=nofollow] { border-bottom: 1px dotted blue !important; }

Got any neat suggestions I’ve missed?

Derek’s got a great career arc. He climbed up the ranks of journalism, covered Congress, and got involved in data-heavy projects. “Computer assisted reporting” is one of those terms that nobody quite loves but nobody’s successfully replaced (though it seems “database journalism” is gaining ground) and refers to collecting and analyzing data in databases. Derek got as interested in the “computers” as the “reporting” and has deliberately pushed his skills and career into software development. (*cough* sounds like a good topic for a quiet blog, eh, Derek? *cough). I look at his GitHub profile today and he’s been busily merging in contributions to his open source projects like his USA Today Census gem.

We started out the day looking around the FEC scraping code. In the States, the Federal Election Commission gives out lots of data from candidates filing required disclosure statements. We tidied up the database a little and then turned to the project for the day, which has just been publicly announced:

Today we’re announcing the addition of paper campaign filings from Senate candidates and two party committees to our Campaign Finance API, which previously had only provided details of electronically filed reports. Now users can request and view the filings of any committee registered with the Federal Election Commission. Unlike House and presidential candidates, current and would-be senators file their campaign reports first with the Secretary of the Senate, who then forwards them to the F.E.C. That agency then scans in the images from the paper filings and makes them available for viewing (an example). While an effort to require electronic filing for Senate candidates hasn’t gotten much traction this year, we have at least made the API’s set of filings more complete. New in the Campaign Finance API: Paper Filings

The scraper was previously ignoring the paper-only reports, but we updated it to recognize and categorize them. The categorization was a huge bit of nostalgia for me: take the noisy and sometimes inconsistent provided categories and map them onto a standard set of database categories (form_type, in the screenshot in the announcement).

When we ran the scraper, it would complain and halt each time it reached an unknown category. We’d add that to a mapping table and restart from that point, but it was frustrating to have to keep an eye on it. So we set the scraper to ignore records that it didn’t have a mapping for and warn about the problem. We set the scraper to run (and hit the amazing Shake Shack for burgers) and came back to find a list of missing mappings. After adding that, we ran the scraper again to fill in the missing entries.

This worked because the scraper only added entries it didn’t already have recorded. The term for this is idempotency, and it’s useful from the level of individual functions up to large, fairly complex programs like web scrapers. Every program fails, having an idempotent approach to the problem means you don’t have to keep careful track of many types of failure because you can fix things and re-run your program without worrying about duplicate records or updating things twice.

Derek had to run off to catch a train, so I dropped in on Dan Berko. He was on one of the Post’s several other “web innovation” teams while I was there, so we helped each other with code occasionally but didn’t spend a lot of time coding together.

The New York Times has large and well-maintained internal tools for reporters and editors. The reporters have a CMS for writing stories and the editors have a budgeting system for planning what goes where in the paper. We improved communcation between these two a bit, so the budgeting tool could refer to a story in the CMS and pull metadata from there instead of requiring an editor to re-input it.

The UI was simple: if the editor links a story, several fields should be grayed out and a checkbox should indicate the link. If the editor unchecks the box, the link is broken and the fields become editable again. This started out with two code paths – one for linking a story, one for unlinking – and making sure on pageload that the UI was in the proper state. We’d barely started writing that when we saw it could be implemented even simpler:

 
disable_if_linked_to_cms: function() {
  var checked = $('asset_cms_id').checked;
  ['asset_home_status_id', 'headline', etc.].each(function(id) {
      $(id).disabled = checked;
  });
},
document.observe('dom:loaded', function(){
  Event.observe('asset_cms_id', 'click', Budget.disable_if_linked_to_cms);
  Budget.disable_if_linked_to_cms();
});

When the checkbox is checked, all the form fields are disabled. When it’s unchecked, they’re not. The code runs on pageload and anytime the checkbox is toggled. I really liked this bit of code: we started out writing the simplest thing that came to mind, but soon we realized it could be reduced. The resulting code probably elicits a “So what, it’s not doing much?” reaction, which is far better than the previous “Now, let’s see, what’s this doing?” we would’ve had at first. The sign of the best code is that you immediately understand it, not that you have to stretch yourself to follow its solution.

Luke did well this week as he started to build out his combat system, including party members who (unless he’s hacked it in, but it doesn’t look that way) respond to locations and enemies. It’s a start at gameplay, though your choices are pretty limited so far. I like the idea in the blog post about having party members you can configure but not directly control, it reminds me of Ogre Battle.

Also, he’s started the storyline, including “the fact that the bear can’t seem to find you is important”. When you play, the bear enemy appears to be completely incapable of attacking you. Just for wild-ass guessing’s sake, I’m going to guess the player starts as some kind of ghost. We’ll see how I did when the storyline takes shape.

Jim… continued to work on the controllers for his web framework, as he did last week. I felt like this:

> Application creates CommandRouter and TemplateDisplay instances

I’m very happy for it.

> Application retrieves Events from the CommandRouter for the current Post data (usually none)

So, it’s like how PHP populates $_GET and $_POST for you before invoking a script, but redundant.

> Application retrieves Events from the CommandRouter for the current URI

Events are sounding awfully general purpose. Wasn’t this a game? I feel faint.

> Application loops through the current events (events can be added while processing other events) and calls the CommandRouter->ProcessEvent() method for each one

It’s a generic message queueing system. The world is going dim…

> CommandRouter creates any necessary controller and sub-display instance for the event and passes all data to any module event handlers registered for the current event

It is pitch black. You are likely to be eaten by a grue.

> Created controllers determine models to use which add their data to the relevant Display objects (which are further applied to the TemplateDisplay instance)

No, really, a grue. It eats developers who don’t finish games.

> Application renders the TemplateDisplay

The grue dies of boredom.

Jim mentioned he liked the Mortal Kombat drawing last week. I guess he didn’t click on it. It’s a link to a great article, which includes:

3. DON’T ROLL YOUR OWN TECH IF YOU DON’T HAVE TO

There are pros and cons to writing your own engine. But ask yourself, do you really have to? Is what you want to do impossible to do with what’s already out there or would you be reinventing the wheel? Of course, if you write your own engine you can make it just perfect the way you like it. But be honest, how often do you ever get past the engine to the game itself? Do you find yourself making game engines more often than you do games?

Bring the thunder, Jim.

Aidan and I hacked away at adding a feature to a Web App That Shall Not Be Named. User authentication was handled by Facebook or Twitter, so the feature was to fetch and list their friendships from their social network. The code in app/models/user.rb was straightforward, taking an object from authentication:

 
# Update the database with the user's current list of friends.
# Compares a fresh list to the existing list to minimize database writes.
def update_friends(auth)
  # fetch current list of friendships as hashes
  provider_friendships = auth.retrieve_friendships
  # load already-stored friendships to hashes
  stored_friendships = friendships.send("on_#{auth.provider}").map { |friend| {:id => friend.friend_id, :name => friend.friend_name} }
 
  # compare the two lists of hashes
 
  # add any new friends to the database
  (provider_friendships - stored_friendships).each do |friend|
    friendships.create!({
      :friend_id => friend[:id],
      :friend_name => friend[:name],
      :provider => auth.provider,
    })
  end
 
  # delete any removed friends from the database
  (stored_friendships - provider_friendships).each do |friend|
    friendships.destroy_all(:conditions => { :friend_id => = friend[:id], :friend_name => friend[:name] })
  end
end

That code isn’t very complicated, but it’s worth noting before you read the next snipped that we were both new to the Facebook/Twitter APIs and spiking functionality rather than trying to write production quality code.

Keep that in mind as you read the code we added to the authentication object. It works, but it could definitely be better written (left as an exercise to the reader, because, well, we only spiked, so I haven’t actually written a better version to present).

 
def retrieve_friendships
  if provider == "facebook"
    # made the Facebook API call
    user = FbGraph::User.me(self.data['credentials']['token'])
    # map it to an array of hashes
    return user.friends.map do |friend|
      # check against the db to load their id
      existing = Authentication.where(:uid => friend.identifier, =
:provider => 'facebook').first
      { :friend => existing ? existing.user : nil,
        :provider_id => existing ? nil : friend.identifier,
        :provider => 'facebook' }
    end
  elsif provider == "twitter"
    # Returns only 20 users at a time - is it important enough at this = point to add all the code for iterating? (aidan)
    # return Twitter.friends(nickname).users.map{ |friend| Hash[ :id => = friend.id, :name => friend.screen_name, :provider => 'twitter' ]}
    return {}
  else
    raise "Refactor me"
  end
end

Having the authentication object test what provider API it should be loading from is an obvious code smell, but it’s not the one I want to talk about.

There’s an implicit type in use. Aidan and I referred to it as a FriendHash, it’s a normalization for comparing friendship objects from the app’s database and from different API calls. FriendHash would be a data class with just a smidge of validation code to account for all three attributes.

I’ve noticed these sneaking around code in the interfaces between objects. I don’t quite have a name for them, but I realize it touches a bit on design by contract. The part that smells to me is that this code is spread amongst many places that need to be updated in sync with each other (not that this is terribly better if you have a class for it, but at least you can grep for references to it). I’m not a big fan of data classes, so I don’t love this solution.

What really frustrates me is that I’ve got enough experience to have a tingling intuition when I am attempting to re-solve some problem that surely someone else has analyzed and offered solutions to, but my resesarch hasn’t turned anything up.

So I’m posting to hear what folks think of this situation. Have you read anyone writing about these quiet but complex data types at interfaces, or do you have ideas for how to address them?

I’ve been increasingly dissatisfied with it since Rackspace bought the company. They added a monthly fee on top of storage costs for no reason, stopped issuing source code to a standalone program for retrieving data, performance hasn’t kept up with my growth in usage (open dialog to choose which files to back up, get coffee), it does an 80MB download on every startup, backups of even a few K finish with a tens-of-megs metadata upload, and now they plan to reduce client-side functionality.

2. New Web-based User Interface

In the spirit of flexibility and accessibility, the next release will have the configuration and setup totally managed via a web-based user interface vs. one that is incorporated into the computer’s operating system. In one simple interface, you will be able to manage every user and computer in a secure web experience that is familiar regardless if you are managing a Windows, Mac or Linux machine. This aspect of the update has been a huge undertaking for us because we want this to be a great user experience. Luckily, we have some pretty amazing talent that is helping us to build a truly awesome UI. The Road Ahead from the JungleDIsk blog

While JungleDisk used to sport "you don't have to trust us" encryption, this new UI means they'll need to keep a list of all of your files. And as much as I love the web, the desktop UI to JungleDisk is already painfully slow and clunky; the web adds network latency, browser overhead, and a much-reduced selection of widgets.

This is the last straw for me. After surveying the field I've settled on tarsnap. I'll write another post with more information on why I chose it and how I've set it up (it's a pretty low-level tool) when I've finished the big re-upload, cleaned up the remainders of JungleDisk, and settled into a regular rhythm.

That calls for a way to sync files, and Dropbox is by far the most convenient way to do so. Dropbox Just Works to copy files between computers, whether on Linux, OS, or Windows. Even an IT department that doesn’t want to install a development suite will probably be OK with it.

But software always calls for version control, it’s essential to keep track of changes. Git happens to be the right version control software for her project, so here’s the steps we worked through to set up her environment.

First, she created her repository inside her Dropbox folder. We’ll call this the dropbox repo.

 
desktop:~/Dropbox $ git init project
desktop:~/Dropbox $ cd project
desktop:~/Dropbox/project $ # create and commit first files

Now she has convenient access to her files across her computers, but there’s a pitfall: if she were to edit and commit to the dropbox repo on two machines that were offline, Dropbox would merge the git metadata in unpredictable ways. Each machine needs its own repository for work done in it, then changes can be deliberately pulled up to the authoritative Dropbox repository for integration. Edits done on a machine without git installed can be left in the Dropbox repo until they can be checked in, which works well enough for this scenario.

Creating a local repository is simple:

 
laptop:~/ $ git clone ~/Dropbox/project
laptop:~/ $ git remote -v
origin  /home/username/Dropbox/project/ (fetch)
origin  /home/username/Dropbox/project/ (push)
laptop:~/ $ cd project
laptop:~/project/ $ git pull # to get the latest changes

The laptop repo shows that it knows about the dropbox repo, and a git pull will bring in any changes. But git push will give a giant error:

 
laptop:~/project/ $ git push
Counting objects: 157, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (152/152), done.
Writing objects: 100% (157/157), 982.50 KiB, done.
Total 157 (delta 44), reused 0 (delta 0)
remote: error: refusing to update checked out branch: refs/heads/master
remote: error: By default, updating the current branch in a non-bare repository
remote: error: is denied, because it will make the index and work tree inconsistent
remote: error: with what you pushed, and will require 'git reset --hard' to match
remote: error: the work tree to HEAD.
remote: error:
remote: error: You can set 'receive.denyCurrentBranch' configuration variable to
remote: error: 'ignore' or 'warn' in the remote repository to allow pushing into
remote: error: its current branch; however, this is not recommended unless you
remote: error: arranged to update its work tree to match what you pushed in some
remote: error: other way.
remote: error:
remote: error: To squelch this message and still keep the default behaviour, set
remote: error: 'receive.denyCurrentBranch' configuration variable to 'refuse'.
To /home/username/Dropbox/project/
 ! [remote rejected] master -> master (branch is currently checked out)
error: failed to push some refs to '/home/username/Dropbox/project/'

This is one of git’s more unix-like error messages, in that it makes sense once you already understand it and happens because git is an incredibly powerful, flexible tool that assumes you know what you’re doing. Let’s walk through the error.

Everything down to the “Total” line is git giving you information about how it’s packaging the data to send between repositories and is normal.

The first paragraph is warning that you’re pushing code to what’s currently being worked on in the other repository (the “checked out branch”). Git can’t know that the other one is just files sitting around – what if it’s some other user’s, and he’s in the middle of editing things? Well, if someone else might be using it, we can’t update the files in the working copy, because it would trash his work. But if git doesn’t do that, the remote repo knows that it’s not at HEAD – meaning the next commit will confusingly create a branch.

The second paragraph is saying “If you’re sure you know what you’re doing, you can configure the remote repository to let you do this.”

The third is saying “Or you can just tell me to shut up.”

Finally, the last few lines are git attempting finish the push. It says “OK, with everything ready, I’m going to push” and then “Wait, I couldn’t push from my master to the remote master” and “you told me to push at least one branch and at least one failed” (in this case, excatly 1 and 1). In short, git is refusing to take the risk of maybe screwing up work.

There are two responses.

The first is to make the Dropbox repository a “bare repository”. That means it will have all the history of the project, but no checked-out, editable copies of the project. That would stop you from conveniently reading the files wherever you happened to be, though. When you have a repository on GitHub, that copy on their servers is a bare repo – GitHub doesn’t have any interest in editing the files and doesn’t want you to have this problem.

The second is simpler, and what you want to do for this situation. Pulling changes is always OK, because git knows you know what you’re doing with the current repo. So, from the dropbox repo:

 
laptop:~/Dropbox/project $ git remote add laptop ~/project
laptop:~/Dropbox/project $ git pull laptop master # note that you have to name what branch to pull from
From /home/username/project
 * branch           master -> FETCH_HEAD

Git reports the path to the repo it’s pulling from and which branches it was able to update, in this case master from what it found on the remote HEAD. Success!

All that’s needed to add more machines is to run the git clone on them and then, from the dropbox repo, add them as remotes for pulling changes.

Dropbox and git play nicely when you need their different strengths.

We worked on FindAPair, a small web app he’s developing with Jeff Powell. Developers can list their city and link to their Twitter and GitHub profiles to find other devs to pair with. (Yeah, this basic idea felt pretty familiar.)

First, Jim and I added some acceptance tests using Steak to test profile editing. Steak is a DSL for acceptance in BDD, but closer to Ruby code than the popular Cucumber. Jim explained he chose Steak because this project does not have a non-technical customer who’d read, verify, and help write specs. Cucumber’s translation between text files and method calls would be unnecessary complexity.

FindAPair uses the wonderful Devise library, so we followed the steps on the wiki to allow people to log in by username as well as email address. It went pretty well, but we needed to do a little tweaking for Rails 3. Of course, we contributed back to the docs. One of the nice things about GitHub is how frictionless it is to share these small improvements.

Another interesting bit of coding was planning the URL structure of the site. The two major resources are users and the cities they live in. Jim wanted to have really simple URLs like findapair.me/jimiray or findapair.me/denver. He didn’t want to overlap the namespaces, but he couldn’t decide between them. As he was explaining the problem I thought back to the early days of the web and suggested putting cities at the root and users at findapair.me/~jimiray. Jim laughed, decided the audience was nerdy enough to remember it, so we committed it.

I’m not going to recount all the little things we did, but it was a fun afternoon building what could be a great resource.

While I was programming with Jim, I was reminded of the importance of studying. I knew a lot more vim commands than Jim, but it’s not because I’m any smarter. When I started learning vim I quickly realized it was a far bigger topic than I could absorb at once, so I only learned enough to take care of my daily routine. After I was comfortable with that, I started reading the vim manual, but I read at most one chapter a week. Read a little, experiment, give it time to settle, refresh my reading in a few days, and then move on to something else. In the decade since then, I still watch for new information about vim, reading the vim subreddit and Twitter accounts like learnvim and dotvimrc. It’s easy to skip over the stuff I already know. It’s easier to earn in small amounts over years than read one book and hope to retain everything.

One result? If there’d been a hundred people I’d have been done right there, but if there’s exactly one person, well, why not say hi? So I sent an email that I think sounded like this:

I realize I’m a complete stranger and the idea of wandering into people’s offices to work with them for a day sounds perfectly ridiculous, but would you like to grab coffee, talk code, do some pair programming?

Perhaps I didn’t come off quite so badly because David said “what the hell” and invited me to meet him for his workday at Traders Coffee.

A side story: I happened to arrive before him and was looking around for him, and couldn’t see him. But I saw a woman surfing Stack Overflow! I thought maybe this was someone David knew and invited along, so I had an awkward conversation where it quickly became clear she had no idea who I was or why I was introducing myself. If I’d been thinking quicker I’d have invited her to join us or at least given a business card, but my generic nerdly social phobia overwhelmed me and I excused myself. If you’re reading this and someone interrupted your Stack Overflow session to ask if you knew a guy you’d never heard of, sorry for the confusion, but if you’d like to grab coffee, talk code, maybe do some pair programming, I’m in town until Friday afternoon.

As David and I sipped delicious lattes, I learned that he’s a mechanical engineer who got into software development around ten years ago as his job changed. He’s self-taught, he cracked the books and started experimenting. Along the way he attended some Construx Seminars. Construx is the software firm started by Steve McConnell, the giant who strides above programmers bestowing books of wisdom and lore, so that gave David a good grounding.

A lot of our conversation was about how David has found it hard to improve his craft without other, ideally better, programmers around to work with. It’s referenced by the community aspect of the Software Craftsmanship Manifesto. Programmers improve their craft by improving each other’s craft.

As an example, David told me about his startup project veloGraf. He and another developer are using graph algorithms to analyze social networks and I soon realized I only know enough graph theory to hang onto the ankles of the conversation. I hadn’t thought about it before, but David pointed out that game tags on NearbyGamers form a highly cyclical graph. Gamers add edges to the graph by listing what games they’re interested in playing and tags self-referentially describe themselves (eg. Dungeons and Dragons is tagged as a an RPG).

I demoed the site’s most computationally-expensive page, which shows gamers who share tags, sorted by distance. This page required some hacking to avoid a gigantic couple of joins that would’ve dragged in most of the rows in the RDBMS, but David chuckled because it’s the sort of problem a proper graph database doesn’t even notice. He’s tracking millions of nodes with many times more edges in order to extract interesting connections. The simple analyses he pointed out flaws in common algorithms that were obvious to him but surprising to me. I managed to give him some fodder for experiments to improve his algorithms, which says more about the value of a little collaboration than it does my graph chops.

David told me a tale of (mild) woe from a previous job. An experienced cowbody coder was developing the core of a product while supposedly mentoring interns. In reality, they were restricted to working on small, ancillary tasks so as not to destabilize the core or distract the cowboy. How do you fix this?

My best guess (share yours in the comments, please – it’s not an uncommon scenario) was to task the interns with adding automated testing. The cowboy will think of it as menial work, but the interns will be exposed to the whole system and have reasons to have conversations with him. And probably better to start with acceptance testing than unit testing to preempt the cowboy from whining that using unit tests to verify that his code works (as opposed to hack and hope and oh yeah, I forgot those two things were related) slows down his brilliance.

There’s a complacency I’ve seen that’s sort of related to the Blub Paradox. When a programmer succeeds at building their first product or two, they often get an panglossian sense of competence. They may think, “I’m good at my job, there’s people telling me I should learn these things, but I’m good at my job without them, ipso facto I don’t need to learn more.” Unless they’re around a noticeably better programmer or especially introspective, they may never realize how much more there is to learn and the possibility of much greater skill.

I think this is where net negative producing programmers come from. I joined a project with an NNPP once. I watched all commits to the repository (everybody does this, right?) and I realized nearly his every commit added a bug or the strong potential for future problems. There was feedback available in the rising bug count and frustration of trying to improve the codebase, but he didn’t have the experience to know that this experience was abnormal. (Or at least improvable.) As I and other developers started cleaning up the code, he recognized what had been happening and decided to reinvest in his career by attending training.

I think this is why the nascent software craftsmanship movement has such potential. Programmers need to make steady improvement, however small, a core value of the profession. Currently there’s a “well, it works” when practice and mindful improvement could be a precept. The likeliest way I see that happening is masters of the craft mentoring apprentices. Every craft has its share of “just get it done”, but it doesn’t have to be the dominant culture.

Update: David has blogged his thoughts on our wide-ranging conversation.

So, it appears that i will have to learn how to use XML for my job. What would you suggest? And by “suggest”, i mean, “talk to me like i’m four.”

I suggest you get a better description of what you need to learn because they didn’t give you enough info. XML is a really general tool used in a lot of different things. (Some of us nerds hate XML and would prefer to see different things in its place, but knowing that doesn’t help you learn XML so I’m not going to talk about it except to finish with a terrible joke.)

XML is a way of marking up data. Imagine I have a catalog card for the (excellent) book I am rereading now:

The Design of Everyday Things
Donald A. Norman
272p, 8.3 x 5.5 x 0.8

Anybody who has ever complained that “they don’t make things like they used to” will immediately connect with this book. Norman’s thesis is that when designers fail to understand the processes by which devices work, they create unworkable technology.
978-00465067107
TS171.4 .N67 2002

It’s a bunch of data. Now, you are a human. You have vast life experience and your brain is a massively parallel neural network, so you are excellent at pattern matching. You know which bits of data are the title or the description or the LOC number, and you’d know this even if you weren’t old enough to have used a physical card catalog (oh god we’re decrepit, why didn’t I use an iPhone or a pop star as an example?).

Computers suck at this. They like to have their data explained to them. XML is one, fairly human-readable way of doing so.

XML would “mark up” the card by labeling the different fields of information.

 
<card>
  <title>The Design of Everyday Things</title>
  <author>Donald A. Norman</author>
  <pagecount>272</pagecount>
  <dimensions units="inches">
    <height>8.3</height>
    <width>5.5</width>
    <depth>0.8</depth>
  </dimensions>
 
  <description>
  Anybody who has ever complained that "they don't make things like they
  used to" will immediately connect with this book. Norman's thesis is
  that when designers fail to understand the processes by which devices
  work, they create unworkable technology.
  </description>
 
  <isbn-13>978-00465067107</isbn-13>
  <loc_number>TS171.4 .N67 2002</loc_number>
</card>

The bits in angle brackets are called “tags”. It’s easy to tell a computer program to look for the <title> tag and save that to a particular field in a database, or display it on a web page, or whatever.

There are also “attributes”, like the ‘units=”inches”‘ on the <dimensions> tag. They’re usually used to describe the way the data is formatted.

Tags can be nested – notice everything is in a <card> tag and the measurements are nested in the <dimensions> tag. Let’s make one more nesting:

 
  <authorlist>
    <author>Donald A. Norman</author>
  </authorlist>

This book only has one author, but there are lots of books with multiple authors, like The Illuminatus Trilogy:

 
  <authors>
    <author>Robert Anton Wilson</author>
    <author>Robert Shea</author>
    <author>fnord</author>
  </authors>

The design decisions of what tags to invent, how to nest them, and how to format data inside them is called a “schema”. XML files should each have a schema to eliminate the Dread Pirate Ambiguity. Is it “Robert Shea” or “Shea, Robert”? Or should we break it out into <first-name> and <last-name> tags, but then how would we deal with a book by Prince, when he even deigns to use an alphabet at all? Either you adopt a schema or you create your own, and hopefully you spend a lot of time thinking about all these weird things that can go wrong because it’s painful to have to change the schema and get everybody to agree to use the new thing.

The only other major concept is “validity”, which means making sure that tags are written properly (no “<auth Robert Shea</author>” or other typos) and used properly (no “<author>Robert Shea and Robert Anton Wilson</author>”). Programmers, being a lazy and untrustworthy lot (and our XML tools often being big pains in the ass), often slap XML files together by hand and generate subtly invalid XML.

XML gets used for a huge amount of stuff nowadays because, despite some shortcomings (the big ones being verbosity, the difficulty of validity, and a lot of crappy programming tools), it’s a decent way of sharing data. And you don’t even have to know much about the schema to look at a file and start pulling data out of it.

If you’ve ever used a feed reader to follow a blog, those feeds it reads are stored in XML. In your browser, pull up my blog feed, click the View menu, and choose View Source. You’ll see the XML describing my recent posts as <item>s and, inside each, the text of each post in a <description>.

A little confusingly, that text is marked up with HTML, which is very similar to XML. XML and HTML are basically brothers, they’re based on the same older work (SGML), except that XML is like the Borg and tried to assimilate HTML in the early ’00s (XHTML) and then the Federation (web nerds) realized that was a crappy idea and set off at maximum warp (let’s go build some awesome things and let the documentation catch up later) in a different direction (HTML5) and soon it’ll be holodecks for everybody (canvas, video, audio tags) and fire photon torpedoes at the Ferengi (Flash). Exactly like that.

So there’s a crash course in XML. I can’t tell you a lot more because I don’t know what your job will have you do with it. Saying they use XML is not much better than saying they use electricity. I know that you’re probably going to take some data from one place, format it, and put it in some other place but that’s a such a vague job description it even applies to those guys with the vacuum trucks who empty latrines (a metaphor that will become increasingly rich, predictive, and pungent to you the more time you spend with XML).

Questions?

The second stop on my craftsmanship tour was last Friday at 8th Light. They’re a local Chicago consultancy that’s active in the software craftsmanship community, especially in building the new Chicago SC group.

I started out pairing with Colin Jones, which was a bit lucky because he’s also a vim user. After I shared the joy of matchit.vim we got down to the task at hand.

8th Light is one of a few companies that do work for Groupon, the local wunderkind. Our task was to write a small script to sync data between services for a small percentage of subscribers, but when a site has over 40 million subscribers, you can’t just hack out something fast and hope for the best.

The work needed was very similar to an existing script, but Colin’s first instinct wasn’t a quick copy and paste and tweak. We analyzed the script to find the points of commonality and extracted them to a superclass, and we grew the existing tests (yes, the one-off script already had tests) to cover the new functionality.

I get the feeling that by the time I finish this tour I’m going to be pretty familiar with every combination of testing and mocking library out there. Last Monday it was Cucumber, this time I dropped into learning rspec and its built-in mocking. I’m glad I experimented with writing my own mocking library a few years ago, I earned a deeper understanding of mocking than one library’s API and was able to explain what I was trying to do and find the right bit of rspec docs quickly.

Every Friday lunch is “8th Light University” and a developer presents on an interesting topic. Colin gave a presentation on Clojure, walking through the basics of functional programming style like map, filter, and the importance of understanding when Clojure will or won’t walk an entire sequence to produce its result (as demonstrated by using (range 10000000000000)). It ended with a little exercise to add up all the numbers between 1 and 1000 that are evenly divisible by 3 and 5, my solution wasn’t too bad, though I should’ve used reduce in place of filter/apply:

(defn evenly_divisible? [x] (or (= 0 (rem x 3)) (= 0 (rem x 5))))
(print (apply + (filter evenly_divisible? (drop 1 (range 1000)))))

That refresher of Lisp syntax and FP came in handy immediately after lunch, when I paired with Micah Martin. He was working on adding a Clojure interface to the GUI toolkit Limelight.

We refactored some Java and Clojure to remove some code duplicated logic for locating and loading GUI elements (“players”). I don’t really have any experience in Java, but it only took me about five minutes of removing duplication before I ran into hassles with the type system.

As soon I was distracted by the interesting problem of structuring code in an unfamiliar language, my hands reverted to use vim commands. We were using IntelliJ IDEA on OS X, so I flailed a bit, at one point griping “I feel like I’m wearing mittens”. It’s disorienting to lose tools I’ve tweaked and practiced with almost daily for a dozen years.

The Java started like this:

 
public void recruit(PropPanel panel, String playerName, CastingDirector castingDirector)
{
  final Scene scene = panel.getRoot();
  final String scenePlayersPath = scene.getResourceLoader().pathTo("players");
 
  final boolean couldRecruitFromPScene = recruitFrom(panel, playerName, castingDirector, scenePlayersPath);
  if(!couldRecruitFromPScene)
  {
    final String productionPlayersPath = scene.getProduction().getResourceLoader().pathTo("players");
    final boolean couldRecruitFromProduction = recruitFrom(panel, playerName, castingDirector, productionPlayersPath);
    if(!couldRecruitFromProduction)
    {
      boolean couldRecruitDefaultPlayer = recruitFrom(panel, playerName, builtinCastingDirector, BuiltinBeacon.getBuiltinPlayersPath());
    }
  }
}

I found this code really hard to read, in part because there were nested conditionals when it’s really doing the sequential work of trying different paths. It’s a bad application of single exit point because Java is a garbage collected language. It’s OK to have multiple return points from a function because you don’t have to worry you’re leaking memory. So I rewrote it as:

 
public void recruit(PropPanel panel, String playerName, CastingDirector castingDirector)
{
  final Scene scene = panel.getRoot();
  final String scenePlayersPath = scene.getResourceLoader().pathTo("players");
  final String productionPlayersPath = scene.getProduction().getResourceLoader().pathTo("players");
 
  if (recruitFrom(panel, playerName, castingDirector, scenePlayersPath))
    return;
 
  if (recruitFrom(panel, playerName, castingDirector, productionPlayersPath))
    return;
 
  recruitFrom(panel, playerName, builtinCastingDirector, BuiltinBeacon.getBuiltinPlayersPath());
}

There are fewer variables and I moved their initialization together, so your eye picks up the parallel structure and just focuses on the differences.

Of course, that’s not just parallel structure but repeated code in the conditionals and call to recruitFrom(). I fell into Ruby syntax trying to explain what I wanted to do as the next step of refactoring:

 
public void recruit(PropPanel panel, String playerName, CastingDirector castingDirector)
{
  final Scene scene = panel.getRoot();
  [
    [scene.getResourceLoader().pathTo("players"),                 castingDirector],
    [scene.getProduction().getResourceLoader().pathTo("players"), castingDirector],
    [BuiltinBeacon.getBuiltinPlayersPath(),                       builtinCastingDirector],
  ].each { |path, director|
    if (recruitFrom(panel, playerName, director, path))
      return;
  }
}

Now the code doesn’t have any extra variables or repeated method calls. If the recruitFrom signature changed, there’d be exactly one place to change it.

Alas, it was not to be. I couldn’t simply bundle a player and a canvas in an array of tuples because they were of different types. Micah started typing a simple inner class to gather them together, but I vetoed it. Even though it would removtge duplication, I didn’t think it worth the tradeoff of added complexity of a new, data-only class. We left it at the previous revision.

Refactoring isn’t a blind set of tools to apply while (code.has_refactoring_possible?) because they’re about managing complexity by moving it to explicit places without duplication. You can spend hours smoothing things out only to be left with a stubborn bit of complexity that you can leave a warning on rather than spend a disproportionate amount of time trying to get that last bit. I find that when I do this, a future change will often expose what the real trouble is and I can advance the refactoring then.

All in all, a great day at 8th Light. I’m still looking for places to visit in Chicago before the end of the year, so please check out my calendar and suggest places here or on Twitter.

I chatted with Corey about his experience and he encouraged me (and every other developer, really) to visit and learn from as many software developers as I could. To this end, he and Scott Parker introduced me to folks at Obtiva and 8th Light, two local consultancies that are part of the software craftsmanship movement, with the encouragement that I visit places even before I leave Chicago.

Obtiva

I’ve gotten my programming tour started. I spent yesterday at Obtiva pairing with Andy Maleh on some client work. We tracked down and fixed a bug, added a straightforward feature, and added a feature that exposed a hidden assumption in the spec and required some refactoring.

This was the first day I’ve spent pair programming and I really enjoyed the experience. There were some tools I hadn’t used, and Andy’s ability to answer a question, give an example, or push me to try it (“Here, you drive”) at the exact time I was wondering how to do something made for quick learning. We traded off responsibilities smoothly, with one of us writing the implementation and the other watching for typos and thinking of different possibilities. There were a few moments when I felt like I had a second brain because I had Andy’s knowledge to draw on or because he pointed out a problem I wouldn’t have noticed for a few minutes because I was heads-down on the current task.

Cucumber

One of those new tools I saw was Cucumber, a test framework for behavior-driven development and acceptance testing. You write your tests in lightly structured English and then write code to implement each step.

I’d seen a lot of talk around Cucumber but ignored it because I’m happy to write my high-level tests in Ruby. I don’t have a non-technical customer to show them to, so I didn’t see a benefit. I’m very glad to have gotten a little experience using Cucumber. It forces you to write a DSL for testing your app, with small, well-named methods to encapsulate each step of the process. It’s a great exercise that I’d recommend for anyone who writes automated tests (which is all developers, right? *cough*).

Dehumanize

As we were tidying up some Cucumber tests, I noticed a bit of repeated code that transformed an English description like “Phone number” into a column name like “phone_number”. We were about to abstract it to a small method, when Andy mentioned he’d submitted a small patch to Rails a while ago that was not accepted. Perhaps inspired by the recent accomplishment of a former coworker, I said, “Why don’t we package a gem?”

So that’s just what we did, creating the dehumanize gem for this snippet to perform the inverse of ActiveSupport::Inflectors#humanize. Jeweler made it the work of a minute to package and publish; we spent more time retrieving Andy’s lost login to publish to RubyGems.

Aside from all these new toys, we had a nice running conversation on the size of commits and using feature branches. My tendency is to make much smaller and more frequent commits than Andy, and to break work off into branches much more often. Look for a blog post about it in the next few days.

Are you in Chicago?

If you’re in Chicago now and would like to spend a day pair programming with me, please contact me. There’s a lot to learn from each other, and it’s a lot of fun. At the least, you can follow this blog’s RSS feed or me on Twitter to see how it goes.

Spreadsheets fascinate me. Lisp proponents point out that “code is data” but in a spreadsheets data is code. Formulas inhabit the cells of the data they represent, a much closer pairing than code files on the same drive as data files.

Spreadsheets are used in institutions of every size, from businesses to government agencies and all the individuals along he way. They were the first killer app and are nearly as ubiquitous as desktop computing. I suspect that most small businesses would be out of business in weeks if they lost the key spreadsheets that get forwarded between office workers and, perhaps, only one employee is allowed to update a master copy of. (This is part of why I’m not surprised that DropBox is doing very well.)

Errors are endemic. I found a nice roundup of research into spreadsheet errors and, even in constrained experimental conditions the cell error rate is above 1%. In a spreadsheet with 100 cells of data, there’s probably at least one error; if there are formulas the rate jumps to 5-15%.

Sometimes this disturbs me, to think of how many errors are creeping along in everything I rely on. How many numbers could I type before I transposed two digits without knowing? Formulas are small, untested programs written in a haphazardly designed language by people who have not studied programming. The design of the software makes mistakes as trivial (for example, click a formula term, change your mind, hit the arrow keys, and you’ve changed the cell the formula is reading data from). It’s astonishing that so many institutions rely on something so unreliable.

But on reflection I think it’s a sign that institutions are more robust than they might appear to first glance. The lights are still on, police still go to the right neighborhoods, products arrive for store shelves. I’ve seen a few attempts to make more rigorous spreadsheets, with data typing, validity checks, embedded programming languages. But I don’t see people lining up to pay that overhead, the benefits they would bring aren’t worthwhile, or at least the losses they would avert are not conspicuously attributable to the lack of those controls.

Thanks for Chrys Wu for the Nation story.

Could anyone recommend a good feed reader for Linux? I need to be able to organize my ~160 feeds in folders, read an entire feed or set of feeds without having to click incessantly, and get already-read items out of the way.

Since I wrote that, tagging has replaced folders, I have over 400 feeds, and I’ve adopted Newsbeuter for reading my feeds. It’s a great console-based feed reader that lets me zip through my feeds with vim-like keybindings. I’ve included my config file at the bottom of this post.

I’ve found it most useful to tag my feeds by how often they update. I’ve found five broad categories:

Personal

Feeds related to my projects and the blogs of friends and family. I want to read all of these several times per day.

Daily

Mostly comics and commentators that update 3-7 times per week, but usually only once per day. These are time-insensitive entertainment, so I can let them build up for a few days or weeks before I go through them.

Irregular

This is the vast bulk of my feeds and update between once per week and once per year. When I like a blog and enjoy reading its archive, I’ll follow it to keep an eye out for new posts. I could read this as infrequently as every month.

Noisy

I only follow a handful of blogs that update multiple times per day and I’ll read them every day or few days, skimming most of the items. Usually any blog that updates this frequently has a pretty lousy signal-to-noise ratio, so they don’t tend to stay in my feedlist for more than a month or two at a time before I get fed up.

Aggregators

Social news sites, usually I read them once in the morning and once in the evening. Sometimes more often if I’m procrastinating.

I’ve found it far more useful to tag my feeds by these reading habits than by their topic. I can sit down to read whatever my current time and attention allow rather than wading through all of it at once.

There’s only one feature I’d like from Newsbeuter. When I travel, I may be offline for several days and only have a few minutes of connectivity when I do get on. I’d like to leave a script of some kind running on a shell account somewhere to collect feed items to quickly spool to me when I’m online. Alternately, I’d be happy running newsbeuter on a remote host if I could still open tabs in my local browser as usual.

Here’s that config file:

# no automatic reloading
refresh-on-startup no
auto-reload no
reload-time 360 # minutes

# reloading
reload-threads 4
reload-only-visible-feeds no
show-read-feeds no
download-retries 4

# notification
notify-screen yes
notify-xterm yes

# storage
max-items 200

# external programs - ff is a firefox wrapper
browser ff

# display
article-sort-order date-desc
feedlist-format "%S%n %11u %t"
articlelist-format "%D %f %?T?;%-17T; ?%t"
datetime-format %m-%d
color background white black
color listnormal white black
color listfocus black white
color info black white
color article white black

# interface
confirm-exit yes
bind-key k up
bind-key j down
bind-key O open-in-browser-and-mark-read

The problem is, about 1/3 to 1/2 of them [design patterns] were basically cover-ups for deficiencies in C++ that don’t exist in other languages. Although I’m not a huge Perl fan anymore, I have to admit the Perl community caught on to this first (or at least funniest). They pointed out that many of these so-called patterns were actually an implementation of Functional Programming in C++.

I’ve had the ranty discussions about how the singleton could be better known as globalton typical of any developer who’s heard that joke:

I’ve never understood why people bother with all that mucking around with instances. Just tag the object initialization on to the end some startup script like the database initialization and add ‘global $map;’ anytime you want to access it. It’s a global variable; most languages have very succinct idioms for managing those, unfortunately.

Singleton has a very small niche managing some hardware access and then is abused approximately all of the time as a noisy version of global variables. Seeing this over and over kept me thinking about whether design patterns are indeed mostly a standard set of tools for working around language deficiencies. (ObSlander: It’d certainly explain the Java community’s fixation on them.)

I’ve had an urge to reread Design Patterns and work through them in a high-level language like Ruby or, better, JavaScript.

But it turns out, as usual, that Peter Norvig is way ahead, with a 1998 presentation on design patterns in dynamic language noting that in some languages design patterns are simply invisible. Unfortunately he didn’t leave behind a detailed pronouncement of opinion as to the value remaining in design patterns when a language invisibly supports the most common patterns at the least, so I suppose I still have the exercise ahead of me.

The standard improvement is to use fixtures, a file with some example data, that your tests can load by giving the name of a fixture. Here’s one of the fixtures for a gamer:

alice: # alice is heavily involved
  id: 1
  mail: alice@example.com
  handle: alice
  password: red queen
  teaser: Follow the white rabbit.
  profile: Not so plain, after all.
  homepage: http://www.example.com
  jabber: alice@example.org
  location_id: 3
  last_login: 2006-10-30 11:56:08
  created_at: 2006-10-30 11:56:08
  location_at: 2006-10-30 11:56:08
  tag_text: dnd, wod
  mail_on_message: true
  mail_on_nearby_discussion: true

This is straightforward, but location_id is the id of another fixture in another file. As you can imagine it’s pretty easy to get them out-of-sync and break some tests. I end up leaving little comments in the YAML to remind myself of the cross-reference.

It’s awfully tempting to reuse fixtures from test to test. Maybe Alice was written to test logging in with a username and password and next I’m writing the test of logging in with an email address and a password. I could copy and paste Alice to another fixture to reuse in that test, but as a programmer I have a pathological aversion to copy and paste. Alice is going to get reused (most of those fields are optional and were used by very different tests), and after this happens a few times it’s hard to change a fixture without breaking an unrelated test.

I’ve gotten a lot of use out of fixtures as I’ve developed the habit of testing all my code, but reuse has made my tests somewhat brittle. After a bit of research, I’ve moved over to using factories.

Here’s the factory for a gamer:

 
Factory.sequence :handle { |n| "Gamer_#{n}" }
 
Factory.define :gamer do |g|
  g.handle { Factory.next :handle }
  g.mail { |m| "#{m.handle}@example.com" }
  g.password 'secret'
end

Only the essential, required fields are listed. Using this simple template, the tests themselves specify the values of the fields they’ll be testing.

 
def test_login_with_valid_username_and_password
  gamer = Gamer.create :handle => 'Alice', :password => 'red queen'
  ...
end
def test_login_with_valid_email_and_password
  gamer = Gamer.create :mail => 'alice@example.com', :password => 'red queen'
  ...
end

Ideally, changing a fixture wouldn’t break any tests at all. And factory_girl (what I’m using for factories in Rails) makes associations easy:

 
Factory.define :post do |p|
  p.association :discussion, :factory => :discussion
  p.association :poster, :factory => :gamer
  p.created_at Time.now.utc
  p.body "Post Body"
end

Now my tests include the specific data they care about, making them easier to understand and improve. And they don’t interrelate, so they’re much more reliable. I do still use fixtures, but now they’re exclusively for tests that need to deal with real-world examples.

Leland on GitHub

Features:

• Shapes: rectangule, hexagonal (‘circle’), freeform (‘free’)
• Variable hex sizes
• shape() to calculate distance in hexes
• highlight a hex
• set a border on a hex
• color a hex
• add text to a hex
• I swear it used to allow setting a background picture, but I don’t see that functionality so maybe I dreamed it or lost it when I misplaced this code for four years.

Known Issues

The code has ugly styling (tabs, odd spacing, no optional braces) – sorry, I was young(er) and stupid(er).

There’s a fair amount of temporal coupling in there – functions that you can’t call before other functions, etc. I hadn’t heard of the command pattern or closures yet.

There’s a bug in the ‘circle’ shape (that the following code exercises):

 
<?php
 
require 'class_hex_image.php';
 
$radius = 3;
$map = new Hex_image;
$map->set_color('bg', 255, 255, 255);
$map->set_l(20);
$map->set_shape('circle', $radius);
$map->start();
for($x = 1; $x <= $map->xsize; $x++) {
    for($y = 1; $y <= $map->ysize; $y++) {
        $map->text($x, $y, 'black', $map->steps($x, $y, $radius, $radius));
        // nice for getting a feel for coordinates
        //$map->text($x, $y, 'black', "$x,$y");
    }
}
 
header('Content-Type: image.png');
$map->finish();       // calls imagepng()
 
?>

Future

One of the guys in #bbg on irc,freenode.net is building a browser-based game with a hex map and I remembered this code was squirreled away somewhere on my hard drive. I have no plans to fix, improve, or in any way maintain this code. I figure I may as well release it so someone can do something with it.

License

I release this code to the public domain. You may use it for any purpose with or without attribution. It would be polite to mention my name and let me know you come up with, but it’s not at all required.

The blog was started by Peter Christensen and has one or two other contributors. With two Peters involved, here’s the handy guide to tell us apart: He’s the guy in Chicago named Peter who’s building a startup around an unconventional web-based game and I’m the guy in Chicago named Peter who’s building a startup around an unconventional web-based game.

I was reading through the blog of one of the GitHub cofounders and an article caught my eye because it described a process very similar to the one that I’ve heard from people who don’t use any version control:

You start your project in a directory named working. As you code, you try to write one feature at a time. When you complete a self-contained portion of a feature, you make sure that all your files are saved and then make a copy of the entire working directory, giving it the name snapshot-0. After you perform this copy operation, you make sure to never again change the code files in the new directory. After the next chunk of work, you perform another copy, only this time the new directory gets the name snapshot-1, and so on.

I like this because doesn’t insult anyone for using a process like this (as I’m occasionally tempted to do). Instead, he sees it as the first step toward the design of a distributed version control system (DVCS) to manage the process. He explains how a version control system encapsulates all the best practices people have learned from decades of coding and collaborating into a reliable software tool.

The following parable will take you on a journey through the creation of a Git-like system from the ground up. Understanding the concepts presented here will be the most valuable thing you can do to prepare yourself to harness the full power of Git. The concepts themselves are quite simple, but allow for an amazing wealth of functionality to spring into existence. Read this parable all the way through and you should have very little trouble mastering the various Git commands and wielding the awesome power that Git makes available to you.

For anybody still on the fence about whether version control is worthwhile, give The Git Parable a read to find out why most developers consider it a necessity. And even if you’re already familiar with version control it’s worth a read to get the basic concepts behind git (like that a commit it not a changeset, as in svn).

If it gets you curious to learn git, the free book Pro Git is the one book you should read. It combines straightforward, readable prose with clear diagrams:

Because Git uses a simple three-way merge, merging from one branch into another multiple times over a long period is generally easy to do. This means you can have several branches that are always open and that you use for different stages of your development cycle; you can merge regularly from some of them into others. Many Git developers have a workflow that embraces this approach, such as having only code that is entirely stable in their master branch — possibly only code that has been or will be released. They have another parallel branch named develop or next that they work from or use to test stability — it isn’t necessarily always stable, but whenever it gets to a stable state, it can be merged into master. It’s used to pull in topic branches (short-lived branches, like your earlier iss53 branch) when they’re ready, to make sure they pass all the tests and don’t introduce bugs.

This book is the best because Chacon finds the right balance between day-to-day interaction examples and exposing the underlying data model. It finds the right balance between “Don’t worry about this magic here” and “and now a discourse on directed acyclic graphs”.

And once you’ve read it, stop. You don’t need to read anything more about git, go forth and code.

Also Considered

The most common negative here is that a work is out of date: Git went from having almost nothing besides man pages from 2005-2007 to an explosion of introductory stuff in 2008 and early 2009. There were also a few significant UI improvements to git in late 2008 and through 2009 — so anything from before then will have a few confusing odds and ends, which is a problem when learning a system that’s already somewhat complex and opaque.

Git Community Book

A close second, the Git Community Book is only slightly less well-structured.

Version Control With Git

A little disorganized (there’s a lot of “X will be explained a few chapters from now”), but the O’Reilly house style you know well.

Git user Manual

Comprehensive and official, but no handholding, this is largely composed of man pages (…“which read like the terse mutterings of pilots wrestling with the controls of damaged airplanes”).

Pragmatic Version Control Using Git

Well-written and includes a nice reference to git-svn. There’s nothing wrong with this book, but Pro Git is newer and has great diagrams.

Learn Git

Well-written (it’s also largely by the indefatigable Chacon) but incomplete.

Git Internals

Scott Chacon’s first excellent introduction, great but less complete than Pro Git. I haven’t seen the screencast.

Git Magic

Fairly comprehensive, but it feels a bit too conversational and doesn’t explain behind-the-scenes fundamentals.

Git for Computer Scientists

“And now a discourse on directed acyclic graphs.” A short article on git’s data structures.

Git Ready

A friendly blog with bite-sized topics, appears defunct.

Randal Schwartz on Git

For too long this was the only beginner intro to git; by the Hooter’s guy.

Linus Torvalds on Git

A fun hour of Torvalds being rude and occasionally awesome, but not any real introduction.

Using Git Without Feeling Stupid One Two

Brief pair of blog posts giving a basic introduction

Git with Rails Tutorial

Outdated and only about deployment.

Understanding Git Conceptually

Basic info on branching, merging, remote repositories, but that’s all.

GitCasts

An outdated and defunct screencast blog.

Git the basics

A good 130m talk on git, but it’s all the data structures before the hands-on.

…and a few dozen other brief blog posts and talks that aren’t worth linking to.

When you’re using one of these markup languages, you’re using HTML anyways. Because they must be translated to HTML to serve as web pages, at best they’re poor translations of HTML. A lot of them are “parsed” with regular expressions, leading to all kinds of interesting bugs with nested tags. MediaWiki (the engine behind Wikipedia) uses two apostrophes for ”bold” and three for ”’italic”’, and then just lets you use HTML for all the cases where these things break. Most markup languages have some passthrough like this to deal with ambiguity or complexity — so while some markups (not BBCode, which is like HTML in brackets) can claim to be easier to start using, users always end up learning HTML in the end.

MediaWiki actually has a decent argument for custom markup in its ability to build templates and fill in particular fields with data. These custom tags aren’t available in HTML, but there’s nothing stopping a site from adding its own tags (which may or may not look much like HTML) that are compiled down to HTML in the exact same fashion.

Some will argue security, that allowing raw HTML would allow people to perform cross-site scripting and other attacks, or include obscene images, or break the page layout, etc. I’m not arguing for letting users use HTML without filtering, it’s absolutely necessary. It’s difficult to protect against the wide variety of attacks, but being difficult doesn’t mean it shouldn’t be done. There are excellent whitelisting libraries available that remove any excuse about rolling one’s own, like Ruby’s Sanitize and PHP’s HTML Purifier.

If you’re building a site, let your users use HTML (of course, replace newlines with <br> for them). A lot will already know it, there are a million free tutorials out there for them to learn it, and after using your site they’ll have a useful skill.

There are two basic strategies for keeping a changing URL pointing to the same content. Either only part of the URL changes and you look things up by the stable piece (as with the slugs in my previous blog post) or you keep track of old identifiers with pointers to their current identifiers.

On NearbyGamers, tags are used to identify what games players are open to playing. But there are a huge number of variations: the game Dungeons and Dragons might be listed as D&D, DND, D&D 4, D&D 4th ed., or many more (and that’s before spelling errors).

To deal with this, I borrowed a mechanism from Wikipedia. Users can edit a tag to redirect it to another (which may not have a redirect set to a third, to prevent all sorts of annoying problems). In short, the Tag object might just be used for its column pointing to another Tag.

This is functionally equivalent to having a table of redirects. When a request comes in for a particular Tag, I could check a TagRedirect table to see if that tag appears in the from column and redirect the visitor to the Tag named by the to column.

Redirects are one of the ways your app will have to scale to meet users’ activity. Thinking about redirects as an explicit table makes it clear you have options for how to deal with it: you could add a date column and expire old entries, or limit the number of redirects one user is allowed to make (perhaps over time), or limit the number of redirects one entity allows. You could even replace this database table with a distributed key/value store like memcached (or preferably something more permanent).

So allowing users to edit URLs leads to one of three things: either you don’t allow them to edit the real identifier, you maintain a table of redirects, or you accept broken links.

NearbyGamers also uses the third strategy: if a gamer changes their handle, the URL is updated without any kind of redirecting. I considered redirects, but then no one could ever relinquish a handle.

URL design is not easy. Allowing users to influence what a URL will be is often really valuable but adds to the difficulty of designing your URLs. This is one of the interesting bits of computer programming where there isn’t a single answer, you have to pick your tradeoffs to match your needs.