1. All Fields Become Optional

   As your dataset grows, exceptions creep in. There’s not enough research time to fill in all your company profiles, there’s one guy in Guam when you expected everyone to be in a U.S. state, there’s data missing from the page you’re scraping, you have to pull updates from a new source.

   Every field eventually loses that beautiful NOT NULL sheen, your code gets filled up with guard clauses of one kind or another and every <div> in your template is wrapped by an if statement. And this happens to foreign keys, too, so OR IS (NOT) NULL sneaks in and left joins mutate into outer joins.

   This is by far the biggest effect on apps over time. It’s getting to the point that I can gauge age by eyeballing the number of fields that retain their NOT NULL constraint.

2. All Relationships Become Many-to-Many

   Some guy works in DC but lives in Virginia, so he needs two Locations. A new type of incoming email needs to be shoveled out to different feeds. A state has both a primary and a caucus. Someone eventually realizes categories never really were mutually exclusive.

   The modern database paradigm is defined by relations, so of course that's what falls apart as soon as you get an app into production. The urge to hack is overwhelming, fudge in a little denormalization or duplicate a row and the pressure's off for now, but it's like freezing a bottle of water, it always grows and breaks worse in the end.

3. Chatter Always Expands

   All the little oddities that change database schemas affect the user presentation as well. Chatter is the intro and outro text around the content of a page that almost no one ever reads. But it has to be there to explain what's going on, the source of information, why things may seem peculiar, the limitations of the dataset, etc. Add in the difficulty of writing succintly and chatter grows until you burn it all down by rebuilding the app.

And when you do rebuild the app from the ground up, you have your chance to slip some NULLs back on, renormalize your data to have easy one-to-many relationships, and present the data in a self-evident and consistent fashion. Then, about a week later, there's a politician who's a Democrat but running for re-election as an Independent...

Poor scheduling

Couldn’t stay up because work is demanding in the election season, so working late would mean I’d limp through the next week of important coverage. More than that, though, was a bad mistake in scheduling a social outing. I didn’t figure in some of travel time and the friend driving wanted to spend a lot more time out, so instead of having a nice morning out I got home at 7 PM tired and in a foul mood. I was able to relax a bit and get some code done, but it cut my available time by more than half.

Too many new toys

I knew I hadn’t played with encoding video before, so I asked an expert for some tips and tinkered around breaking test videos down into short, easily-transcribed segments. Not coincidentally, this is basically the only feature that works. I started with bort to get a lot of basic functionality quickly, but I had no experience with most of the plugins it provided (and those I otherwise planned to use).

aasm

Differences between acts_as_state_machine (which I started with) and aasm (the successor that bort ships).

haml

Translating scaffolding from erb to haml and a (still-unsolved) bug that causes it to render content twice on one page. (And I’ve even used haml before.)

open_id_authentication/restful_authentication

I wanted users to be able to contribute anonymously or use OpenID to receive credit for their contributions without having to set up an account, and the plugins came configured for conventional user accounts.

resource_this

I had to fix a templating bug, and there’s a fair amount of magic to learn to work with.

rspec

Scaffolded tests needed slow, tedious tweaks to work with resource_this. I think I like the idea of rspec more than the implementation.

paperclip

Sorting out finicky path/url issues, and I couldn’t find a way to mock validates_attachment_presence so several models went unspeced..

Each of these issues ate 30-60m, and there went the time to build. Some plugins (acts_as_versioned, footnotes, random_finders) just worked great, but none was really related to core functionality. Speaking of which:

Core featureset too big

The line between nothing and application includes site graphic design, listing conferences, uploading presentation videos, splitting videos into segments, adding transcripts, viewing all of the these, editing transcripts… I had a lot of features listed as optional, but there was still a long set of base features and each uncovered another one of those frustrations.

No teammates

It would’ve helped a lot to have someone to work with. Not just because it would have taken less time to finish the core features, but for the enjoyment of teamwork and to complement my weaknesses.

All told

I’m glad I spent time I spent hanging out in #railsrumble or on the RailsRumble present.ly. Even if it didn’t directly result in items checked off the todo list, it’s been a fun community to be in, and I was glad to help folks out. I’m eager to see what other people came up with.

I didn’t enter RailsRumble to win, I entered for the excuse to create what I thought was an interesting small project. I’ll pick ConfReader back up in a little while, probably after the election. I still think it’s too good an idea not to build, but I’m a bit tired right now.

1. A graphic designer thought it would be more harmonious to have them the same color.
2. A business guy thought it would increase pageviews.

The worst part is, they’re both right. But they’re assholes because they’re selling out user experience to avoid picking a color or to inflate a not-very-useful statistic. Users are deliberately confused and led to click in circles for their trivial convenience or wrongheaded book-cooking.

This boiled over for me when I was trying to read CopyBlogger. They have some really nice writing, but I found myself opening the articles over and over because I couldn’t see that I’d visited it. I just gave up on reading the site, who needs assholes?

A few days later I remembered that Firefox has a setting in the preferences to override page and link colors. Unfortunately, it doesn’t let me override link colors and leave page colors alone.

The way to do undo the damage these assholes have done to their own site is to add a user stylesheet to Firefox. It’s a styling file that Firefox loads for every single page you visit. In Linux it’s ~/.mozilla/firefox/random junk/chrome/userContent.css. On other systems, search for the file userContent.css, though you may just need to search for the chrome directory and create the file. Here’s what to put in there:

 
p > a { color: blue !important; text-decoration: underline !important; }
p > a:visited { color: purple !important; text-decoration: underline !important; }

This forces all unvisited links in the text of a page to be blue and underlined, and all visited links to be purple and underlined. This isn’t perfect, sometimes the text of a page will be on a blue background, but it works well for nearly all pages I visit. The best part about is is the p > keeps it from changing link colors in headers and other parts of the page that are almost never on white backgrounds.

On Sunday I updated gamers and tags to use ActiveResource. It wasn’t hard, mostly just renames (the ‘view’ action became ‘show’, for example) and removing url_for hashes (:controller => 'gamers', :action => 'view', :handle => @gamer.handle) with much nicer named routes (gamer_path(@gamer)). A gripe I have about ActiveResource is that it has poor URLs, with internal record IDs leaking out. I’ve had nice URLs (/gamers/Harkins and /tags/RPGs) for gamers and tags and I wanted to keep them.

Joshua Schacter gave a presentation on building and growing del.icio.us that included the gems (among many other gems):

• Dont expose your unique id’s to the outside world (php?id=1 etc.) people can scrape through everything very easily. This is why del.icio.us uses MD5 hashes of links instead.
• URLs are prime real estate – respect them

It turns out this is not just possible but easy, though I blew an hour or so finding it (Rails docs still need love). In my Gamer and Tag objects, I just define a to_param method and the URL builder picks it up:

 
class Gamer < ActiveRecord::Base
  def to_param
    self.handle
  end
end
class Tag < ActiveRecord::Base
  def to_param
    self.name
  end
end
 
# and over in config/routes.rb I do the usual:
map.resources :gamers
map.resources :tags, :memger => { :history => :get, :variantes => :get }
# and now gamer_path(@gamer) and tag_path(@tag) work in my views

My controller looks like:

 
class GamersController < ApplicationController
  before_filter :load_gamer, :except => [ :index ]
 
  # The regular index, show, edit, and update actions here
 
  private
  def load_gamer
    @gamer = Gamer.find_by_handle(params[:id], :include => [:location, :tags])
    raise ::ActionController::RoutingError, "Recognition failed for #{request.path}" if @gamer.nil?
    # canonicalize case
    redirect_to gamer_path(@gamer) and return if @gamer.handle != params[:id]
  end

And every action that needs it has the @gamer object loaded from the URL param, with 404 errors handled and mistaken case fixed. There’s a patch that will let you rename the :id parameter. I can’t say I mind if it’s not accepted; I don’t mind calling it :id instead of :handle or :name because I think of it as the public id of the object.

Come back Thursday, I’ll tell you how I made my discussion URLs even more useful.

The terms and conditions are exactly 2,840 words long. At 9 words each, that’s 316 pages. There’s no way to resize the TOS div and, yes, the whole form is abnormally wide.

Between the above problems and the problems in the rest of the signup, I have to guess they failed to test in Firefox. I can see why Microsoft might not want to, but 30% of surfers use Firefox, and it’s a far higher percentage higher for the web professionals who are going to be using this service.

It does get worse: when I go to sign in I type my username and password, click “Sign In”, and am presented with the sign in form with my username and no password or errors. It just silently fails to log me in. I’m not too optimistic about their support, they can’t even keep the url to it from breaking across two lines in the “Welcome to AdCenter” mail.

I like to break down stylesheets so there’s a site-wide stylesheet with global stylings like fonts, the site’s template, and common elements. This is the file that metastasizes on you.

My solution is to break down stylesheets by controller and action, and Rails makes this quite easy:

 
# app/views/layout/application.rb
<%= stylesheet_link_tag *(['global'] + @stylesheets) %>

 
# app/controllers/application.rb
class ApplicationController < ActionController::Base
  before_filter :add_stylesheets
 
  def initialize
    @stylesheets = []
  end
 
  def add_stylesheets
    ["#{controller_name}/_controller", "#{controller_name}/#{action_name}"].each do |stylesheet|
      @stylesheets << stylesheet if File.exists? "#{Dir.pwd}/stylesheets/#{stylesheet}.css"
    end
  end
end

This code automatically loads stylesheets for the controller and the action if they exist. The subdivision allows you to make them as compact and specific as possible. If a partial needs its own code, write at the top:

 
<% @styleshets << "controller/_partial_name" %>

Nice and tidy.

I’d say Vox is basic LiveJournal aimed at an older audience. It’s a blog for people who hear blog and think “Oh, those things that the newspaper keeps mentioning as bad for business and the cable news is so scared of.” People who don’t want to share pictures of their kids with random internet strangers but find their cc list getting unmanageably long. Vox is a blogging site with a surprisingly nice user interface, the first I feel I can recommend to my mom without signing up to do a few hours of tutoring and answering questions.

It really is nice. Six Apart clearly learned a lot from LiveJournal and thought hard about how to make a blogging service to let you bring your relationships with family and friends online with minimal weirdness. Vox is not trying to be an engineering triumph, it just is a user interface triumph.

All is not roses, of course. They’re not fully launching until roughly October. You have to join an announcement list to get an account, I got the impression they’re adding a batch of a couple thousand people every few weeks. The site simply doesn’t work in Safari yet, and IE is iffy. There’s no public bug tracker, so I can’t even guess at how many features are missing or broken, so I can’t recommend anyone sign up until the launch in a few months. Also, if you happen to know any hardcore LiveJournal fans you’re in for a rant about how Six Apart wants to destroy LJ and Vox is some kind of weapon in this covert campaign.

The party was one of a series Vox is doing before launch ostensibly so that Vox doesn’t just grow out of California like most social websites. There was a blurry demo, open bar, and around fifty people at the MCA. I enjoyed chatting with four of the Six Apart employees present (out of seven or so) and around a third of the other guests.

I look forward to seeing launch, and web developers should learn from their very nice blog posting interface.

She owns a good variety of socks which are safety-pinned together anytime they’re not actually on her feet. Her socks do everything paired up. This solves the matching problem and greatly reduces the likelihood any go missing (and at least they do it in pairs). If one should wear out before the other, though, they’re both lost.

My strategy is to own a large number of exactly two styles of socks: black unlabeled athletic ankle socks and black unlabeled athletic crew socks. Sorting is trivial, loss and damage are inconsequential. Wearing mismatched socks is only sometimes necessary on laundry day and is at least unobtrusive. When I notice one type of sock is low, I just go buy another dozen.

What’s your sock strategy?

2007-02-22: My girlfriend would like it to be known that she does not dispose of the surviving complements to worn out socks, they go into a drawer to be potentially matched up with other socks whose mates are lost. This distinction is vital, and I admit my sole culpability in its oversight.

Unicode puts out PDF code charts of each and every glyph so people can have some idea of what they should look like (though the actual display to the user is left up to fonts). If you’d like to see one, the chart for Linear B is a nice little example. Because I didn’t want to mess around with putting dozens of PDFs together, I just grabbed the single 30MB 777-page PDF for the latest version, 4.1.0. (New versions are released as they add more scripts — cuneiform is scheduled for 5.0, yay!)

Looking at the PDF (whether you grabbed the small example or decided to be macho and pulled up the whole thing), it’s obvious that the “one big table per page” approach makes it pretty easy to extract the glyphs. I wrote a program to do this, because it would be stupid and ultimately self-defeating to open the PDF with GIMP to manually select and save each glyph.

First, I used ghostscript to convert the PDF to a series of 600 DPI pngs, one image per page. This took about 39 minutes and 471MB of disk space. (I’ll explain why I converted at such a high resolution in about two paragraphs.)

Next, I wrote a Python script using the Python Imaging Library to analyze every page, locate the glyph table if the page has one, and walk through the table exporting each glyph to its own individual PNG. The 769MB of individual glyph images took about 5.6 hours to rip. And that’s OK! I kept the code simple to read because performance doesn’t matter. That’s an odd thing to say for a program that takes hours to finish, but you only ever run this program once. Note, that’s never an excuse for shoddy quality, because you never run a program just once. Yes, those two statements are at odds. If you don’t see that they’re both true, you have not yet learned the Tao of Code.

Well, there’s one important step elided from the previous paragraph: figuring out which glyph is which so I know what the filename should be. Every glyph is labeled with a four or five-digit hexadecimal number, but of course the label is part of the image rather than text. Converting images to text is known as OCR (optical character recognition) and it’s difficult to avoid errors. In my favor I had images that were big (600 DPI, see, I told you I’d get back to it), error-free, and black-and-white; but a strike against me was that it can’t spell-check a word like “A29C”. After trying a few different OCR programs, I got ocrad working well (with a little tinkering, it often mistook the number 0 for the letter O).

Well, mission accomplished. I’ve ripped 97,715 glyphs from the Unicode code charts to individual PNGs. So what should I do with them all?

I work at Hostway, a web hosting company with about a dozen offices worldwide. I picked up a copy of our logo in vector format from one of our graphic designers and resized it until it had about the same number of black pixels as I have glyphs.

Then, of course, I wrote another script to render a final image. It created a huge new image in which every pixel from the source was a downsampled (because I’m printing at 300 DPI) glyph, and it tries to make sure it puts the darkest glyphs in place of the darkest pixel so I have a (very) rough antialiasing effect to soften the edges of letters. (If it’s not already painfully obvious, yes, I did ascii art as a teenager.) This script immediately crashed and burned.

The problem is scale. The source image is 1618 x 151 pixels, and I’m outputting at 300 DPI: 1618 x 151 x 300 x 300 = 20.9GB. As cool as it would be, I just don’t have that much RAM. I had to break up the image into pieces three feet wide, which is good because that’s also how wide the biggest printer in the office is. And I got to write a great error message: “Can’t create images this big. Reduce max_piece_width or buy more RAM.”

Made of 16 images totaling 310MB (it took 45m), the Hostway logo worked out to be 6 feet tall and 46 feet long. My boss told me it was “too big” to hang up. I told him that was “exactly the point of the whole thing”. I’ll change his mind yet — everyone in the office I show it to loves it, so it’s just a matter of time.

I presented the code at the June 2006 meeting of ChiPy and it was well-received. One attendee told me afterwards it was the funniest, nerdiest presentation he’s ever seen, which I consider high praise. As promised, I’m sharing the source code.

unicode-art.2006-06-15.tar.gz

You’ll have to be a little familiar with graphics and Python to make use of this. It is not a copy of all the glyph graphics, and I will not provide you with them or make posters for you, so please don’t ask. I’d love to hear what about what you do with it, please do something cool and post a comment or trackback.

I know a few folks are dropping by looking for the source I promised to post. It’ll be up later tonight or tomorrow morning — I had to recreate one of my scripts from scratch and I’d like to make sure I did it right so I don’t post broken code. And as all of you at the presentation understand, running any of these scripts takes a few hours. So it’ll be up Real Soon Now.

Thanks again to Fawad Halim for loaning me the use of his laptop for my presentation.

Of course it took a little longer than I expected, but it’s up now.

PHP makes it very easy to create bad URLs like /member.php?id=8. Those are bad because web spiders don’t like to crawl URLs with GET variables, some browsers don’t cache any GET URLs, they expose that you use PHP (when the visitor should never even know), and they’re just downright ugly and hard to remember. I’m going to present a way to build a PHP/Apache site with clean URLs.

Let’s look, line-by-line, at the contents of .htaccess. While writing this article I found a more elegant equivalent in the WordPress code, so I’ll present that here:

# Tell Apache to load mod_rewrite.
RewriteEngine On
# Rewrite URLs for the location starting at /
# Note this is URL location, not a path to your web root.
RewriteBase /
# If the request asks for a file or directory that doesn't exist
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
# send the request to index.php.
RewriteRule . /index.php [L]

The hard part about this change is the shift in thinking. A URL isn’t just a path to a file you FTP’d to a web server, it’s a Universal Resource Locator, an address for information. It doesn’t matter whether your site presents data taken from files, a database, or a random number generator — a web browser requests a URL and knows nothing about where the website gets it from.

With that in mind, let’s look at how your PHP site can take apart the URL to route the request to the right PHP script. Create an index.php that looks like:

 
function url_parse($url) {
        // strip off get vars and anchor tags
        if (strpos($url, '?'))
                $url = substr($url, 0, strpos($url, '?'));
        if (strpos($url, '#'))
                $url = substr($url, 0, strpos($url, '#'));
 
        //remove leading slash and possible trailing slash, store in $url
        if (substr($url, 0, 1) == '/')
                $url = substr($url, 1);
        if (substr($url, -1) == '/')
                $url = substr($url, 0, -1);
        if ($url == '/')
                $url = '';
        $url = explode('/', $url);
        
        return($url);
}
 
$actions = Array(
    "" => "front_page.php",
    "mail" => "mail.php",
    "member" => "profile.php",
    "messageboard" => "boards.php",
);
 
$url = url_parse($_SERVER['REQUEST_URI']);
$action = array_shift($url);
if (array_key_exists($action, $actions)) {
    require($actions[$action]);
} else {
    require("404.php");
}

So index.php takes the first element off the array, and probably uses it to require() specific templates or scripts that do the work with the rest of the $url array. Think of it like a switchboard operator — it sends requests where they need to go.

Your individual pages can array_shift() their arguments from the $url array. In the example above, profile.php would expect a username or ID number, and it can require("404.php"); if there’s nothing there or no user by that id.

More complex nested URLs (say, /messageboard/chat/new_thread) work much the same way index.php works: the base script examines $url and passes it on to the scripts it knows about. In the example above, boards.php can load the requested board or the pages used to create a new post or a new board.

In my next post, I’ll provide a clean URL solution for existing sites that can’t afford to redesign their PHP scripts.

My correction is to Deane’s list of “really poorly-formatted text” he sees over and over — he actually has two lists:

Really poorly-formatted text

• No concept of whitespace
• No awareness of how images should be positioned
• A desire to center every heading, title, and subtitle
• Attempts to “hack” horizontal whitespace with tabs and consecutive spaces
• Overly large tables without headings, captions, or consistent spacing

Really poorly-written text

• No use of headings
• No use of bulleted or numbered lists
• Overuse of underlining and boldface
• Paragraphs that are way too long for the Web
• No awareness of the difference between a line break and a paragraph break
• No awareness of what terms should be hyperlinked, and where they should link

Writing well is hard, especially for a young medium like the web where we’re still figuring out what works on top of the fact that most people never actually learn to write well. I saw a letter a few days ago that read “You have been selected for a process called VERIFICATION. Verification is a process by which we are required to verify certain information you submitted on your FAFSA application.” Formatting is not that writer’s first problem.

Additionally, writing on a computer requires thinking about structure and form in a way that’s wholly alien to writing longhand or on a typewriter. On those, what you see is all you get. What you see on a computer is a representation of some content controlled by generally invisible metadata. Today’s kids are still being taught by the typewriter generation, so it’s going to be at least another twenty years before the kids who grew up with an understanding of this duality start teaching to the kids who won’t believe it could be any other way.

A lot of web surfers have never heard of social bookmarking sites (SBS). Gasp, shock, horror, not everyone is up-to-date on the web 2.0 meme. I put a little introduction to SBS in the hover text for newbies, and deliberately didn’t make it editable through the admin interface so that it wouldn’t get changed into a thousand variations on “OMG CLIK & MAK MY BLOGG POPULER!”. Users can (and do) edit it in the PHP, but as Sociable users are not especially technically savvy it’s uncommon.

More experienced users say linking to SBS is hammy because not every reader uses them, because the sites provide their own bookmarklets and plugins, etc. I don’t disagree. Sociable exists because blog authors want more traffic and SBS are a great way to get it. Sociable exists for the same reason grocery stores put candy in the checkout lanes and Amazon lets you buy books with one click: as soon as readers finish a post they’re reminded they can share it with others, it’s the web 2.0 equivalent of the impulse buy.

I wrote Sociable because I wanted to add those links to the bottom of my blog posts. I thought writing a WordPress plugin would be fun, and other people might like to use it. With all that in mind, let me talk about the graphic design decisions I made.

By default, Sociable only adds icons to posts when they’re displayed on individual pages and not on the front page, archives, search results, or feeds. A whole lot of users set Sociable to display everywhere with predictably bad results. I added this option because I’ve seen blogs that don’t have (or don’t prefer to use) pages for individual posts, just archive pages. I weighed this decision heavily but decided in this case that it’s better to support more users than stop all users from shooting themselves in the foot. (This option is a pretty good analogy for the different design philosophies behind Unix and OS X.)

The other big default is to show all SBS icons, and it’s getting worse and worse. New SBS contact me about once each week to get added, so pretty soon the default is going to pass from humorous overkill into a sprawling mess. (And let me parenthetically tip my hat to the folks who’ve realized it’s intentionally silly.) I’m not sure what to do about this one yet. If I have them all off by default, I’m going to get inundated with mails from people who’ve installed it and rightly say it doesn’t work out of the box. Bad first experience. If I turn on only the most popular few by default, there’s an ugly “rich get richer” effect. I haven’t thought of a good solution and would appreciate advice.

Lastly, I tried to keep the visual clutter down as much as possible in the default CSS. There’s a caption, and then the icons without adornment, period. The CSS even has to deliberately strip away a lot of visual styling that different blog themes would otherwise add. Most of the noise comes from the fact that the sites’ icons are all visually dissimilar with different color palletes. Any random blog is guaranteed that most of them won’t match the blog theme, let alone each other.

At the moment I’m looking at it as a good incentive for blog owners to prune down the number of sites they display, but it probably needs more help. One user had a creative solution to display icons in grayscale until they’re hovered over, and I’ll include that feature (probably in the next version).

I hope I’ve addressed the valid concerns of critics, and I appreciate any advice for further improvements.