Campaign Tracker

I made it possible to browse events by type. I’ve also gotten data to academic researchers who’ve requested copies of this (I think) unique data.

Voters Guide

The 2006-8 Metro Voters Guide is probably the clunkiest app I worked on. Lots of confusion between producing a one-off site and a maintained app, as well as between primaries and generals. 2009 will be better because we’ve settled those issues and we’ve built a nice database schema for dealing with elections as part of the 2008 Presidential race.

Congress Votes

I loaded the new info for the current and previous sessions of Congress, added the ability to view just current members of Congress (as some join or leave during the term), added JSON feeds that power the Apture displays (yes, I know about the NYT Congress API, I’m getting the signoffs to publicize the machine-readable pages).

Staff E-Mail Form

Mostly just maintenance, but some generalization for shared mailboxes.

Challenge Index

Formerly the clunkiest app, we fixed it by refocusing it on local schools (dropping national) and I rebuilt it in Dec 2008 to clean up the code. Then we imported data back to 1996, and it’s a really nice little app. Also the source of my 3rd rule of database apps (watch for the post).

Presidential Endorsements

During the primaries, we tracked endorsements from public figures, mostly politicians. The app is interesting today: the front page only mentions the last few candidates like Clinton, Obama, and McCain because we stopped advertising the candidates as they dropped, but their data is all still in the db. We’ve been meaning to resurface them now that the election is settled, but it’s hard to remember to find the time to be an archive instead of just news.

Sellout Songs

This went along with a cute piece in the Sunday Style section. Fun fact: Python has one constant for pi, the graphic used another, and the example songs used a third. If you play with the app, you’ll see it matches the examples (because that’s the only discrepancy folks might notice). This project is where I started dreading the Sunday paper: because there’s a Saturday edition (the “bulldog” edition), things have to be ready late Friday evening. And print folks are under a daily pressure that has driven them all a bit mad, so they tend to finish projects right up against the deadline. Since this project we’ve managed projects better to avoid the “OK, now that it’s 7 PM on Friday, here’s everything you need to build the web app” problem, and I’ve noticed the web has gotten more consideration in the print newsroom over time.

Campaign Finance Search

Candidates are required to report contributions over $200 to the FEC and we pull that data from them. We were showing state, city, and zip code totals, then I built a search engine to expose the individual records. (And we also track noteworthy donors.) Take a good look at the search page and results table so you know it as you continue reading this list.

Dog Parks

A small app showing off Dog Parks in the DC Metro area. We’ve talked about adding more features, but it’s never climbed high enough up the priorities ladder.

Recipes of the Day widget

The only (technically) interesting thing about this project was preloading a month’s worth of season-appropriate recipes at a time, because it’d grind down a producer to have to update this by hand every day. It’s a nice toy.

Silent Injustice

The FBI used to analyze the content of lead in bullets to match a bullet in a victim to bullets owned by a defendant. Trouble is, the science didn’t really work, and they quietly dropped it without notifying people convicted in whole or in part on this unreliable evidence. I helped put all the case files we could find online. This was a really rewarding project: it may have only mattered to a few hundred defendants and their loves ones, but it meant a lot to them.

Primary Pages

They showed live news and election results on election nights, and our database server struggled a little to keep up that first Iowa night. Postgresql doesn’t have good replication options like MySQL, so we beefed up the hardware before Super Tuesday and have been running great since. On some of the election nights, traffic to the projects server accounted for more than 10% of all WashingtonPost.com traffic.

Primary Exit Polls

Lots of fun breakdowns of data by demographic, location, response, etc. One thing I noticed on this project is that it’s hard for humans to talk about breaking down data in more than three dimensions (and three itself can be tricky). Funny how our visual and spatial skills affect the rest of our cognition.

Post 200

The Post 200 showcases 200 prominent local businesses; apparently it’s been a yearly staple for a few decades. In 2008 we built a site we could continuously update, rather than just drop a little data once a year. I’d still like to grow this app to encompass all local companies, or all publicly traded companies (because more data makes everything better, right?). The Post doesn’t do a whole lot of business coverage, though, so we may not be able to add enough value to make that all worthwhile.

Clinton’s Schedule

In the 2008 primaries, Hillary Clinton supported her claim of being more experienced with her time as First Lady, and eventually acquiesced to the call to release her schedule so people could judge her time. I found out the day before (while chatting at the water cooler) that her campaign would be dropping it off at the National Archives. Seemed like it might be fun to do something with the data, but we had no idea what format it would take. The next morning a courier picked up a CD from the Archives and we saw it in all its glory: a 17,481 page PDF that looked like it had been scanned from the fax of a photcopied inkjet printout. If you look at the news stories written that day (2008-03-19) you’ll notice they all take the form of “What was Clinton doing on this important date?” because reporters couldn’t search, just page through it chronologically. I used 200 Amazon EC2 instances to break up and OCR the entire PDF in a few hours and we put the searchable database online (and it became an AWS Case Study (note: PR people wrote that thing)). The project was an absolute blast: making interesting data available by applying cutting-edge tools on a tight deadline.

Young Lives At Risk

I pulled nutritional data on 11k foods from the USDA (in tilde-delimited format, weird) and it powered an interactive shopping cart. Over the few weeks of this project, the USDA called me nearly daily to chat about nutritional data. I respect their commitment to nutritional data, but by the end of it I just wanted to buy them all puppies so my phone would stop ringing.

Sacred Ground

I didn’t work on the (very nice) Flash app, I updated the Faces of the Fallen app to user-submitted tributes and the “Sacred Ground” templates for Pentagon victims. This looks really nice if you’re surfing from this story on the opening of the Pentagon Memorial, but it’s hugely confusing if you’re visiting from the main Fallen app. It’s one of those last-minute compromises that sometimes happen, and I plan to repair the damage.

Business Glossary

When the CDO market finally had to face reality and cratered, the business dept. asked if it would be possible to build a resource for all the (unfortunately) new terms readers were confused by. Working from the Politics Glossary (gee, eerily similar), we were able to respond with an entire application instead of just a promise to build one.

Thrift Savings Plan data

The iframe near the bottom of the page with TSP market information (sort of a 401k for government employees) is a small app that pulls data from the TSP.

Vote Monitor

Our team kept some empty space in our schedule because we knew the Politics team would surely come up with a last-minute idea for an election app, and they did. After some debate about whether to stay local or open up to national reports, we built a great app to allow people to report their voting experiences. I wish we’d had the idea sooner to better advertise it and work closer with other organizations that monitored the elections. Note for 2010 to-do list…

Obama Transition Donations

Hey, remember that table with donations presidential candidates table from back when? Obama, big on transparency, has released information on the donations to his presidential transition and we’re presenting a fairly familiar view of it. I believe they’ll release the data on January donors in about two weeks, so we’ll be updating then.

Clinton Foundation Donations

Hey, remember that table with donations… yeah, you remember. As part of Hillary Clinton’s confirmation process as Secretary of State, the Obama team required the Clinton Foundation to release previously-private donation information. They grudgingly posted it to their website, which crashed under the load. We scraped what we could as the site was intermittently up and presented it.

Bush Legacy Timeline

Finally, reaching a project that went live two days ago, the Bush Legacy Timeline graphs key indicators against important events. I created the database to store events and helped out with the Flash coding.

Some other projects that I haven’t done much more than bugfix are Local Blog Directory, Recipes, DC Schools, Political Ads, Faces of the Fallen, High School Recruiting, and Inauguration Alerts. I’ve also built a few internal tools to automate or streamline the editorial process, but they don’t make for compelling reading and this post is already plenty long.

In case you’ve wondered why I’ve called the Post the best job I’ve ever had, scroll back up and read it again. Tons of interesting, meaningful projects with more talented professionals than I could manage to list, but let me try so this post doesn’t make me look like a braggart:

The foundation was laid by Adrian Holovaty and Derek Willis. It’s the nicest codebase I’ve ever inherited. Almost every one of the apps I listed is well-designed because of Alyson Hurt. Several of these data sets came from Sarah Cohen. I’ve worked a bit with the developers Dan Berko and Ed Holzinger. Ryan O’Neil is the other developer on the Tools Team who builds all these fun apps with me, and he occasionally saves me from my bugs and design decisions. There’s also all all the news producers who created and managed apps: Jason Manning, Sarah Lovenheim, Paul Volpe, Andrea Caumont, Amy Kovac, Tanya Ballard, Amanda McGrath, Jacqueline Dupree, Alicia Cypress, and Amanda Zamora. Last, it’s been possible because of great bosses Bob Greiner, Karl Eisenhower, Jim Brady, and Dave Baker.

More TK.

One of the most important articles I’ve read on the c2 wiki wasn’t about a programming technique. It was PlayHurt. The core debate of the page is whether software developers can produce good code even when their hearts are not in it. Because development is a creative mental activity, it’s profoundly affected by the developer’s mental state. When I first read the page, one comment on it really caught my attention:

Is it possible that few of us have ever done anything but programming for a company while playing hurt? Companies seem to put most programmers into unproductive situations most of the time. Without properly supporting a technical developer, then developers PlayHurt and probably fail (the 95% of software projects fail rule again). For example if the marketing is not in place and we don’t have and never will have any real customers then we are playing hurt no matter how good a job we do on the code.

I was working on a product that was unrelated to the rest of the company’s business. It had originally been built in a hurry by inexperienced developers in an unfamiliar language, and no one inside or outside the company had a complete understanding of the product (self included). The CEO set the goal to increase sales 10,000% percent by the year’s end without creating a marketing budget (10% growth would have been lucky). The company culture was built around churning recent college graduates, so there were low standards and no professional criticism.

I don’t often read things and think they speak exactly to my situation, but that comment sure did. It put a name to what I was feeling. I felt terrible because I didn’t understand why I didn’t want to go to work in the mornings and why I had to push myself to fix every bug and complete every feature. I didn’t have the perspective or experience to recognize all the things I just listed that had gone wrong, so I felt like a failure because I couldn’t develop on command. I found some of that understanding on the wiki. I couldn’t Play Hurt anymore. I was just plain tired of being hurt. I quit that month.

If you want to write good code, you can’t play hurt. If you want to be proud of your work, you can’t play hurt. If you want to feel you’re making something of your life, you can’t play hurt. The best work is powered by passion and meaning, not obligation. Play hurt if you’re getting through the occasional frustrations that every job has, if you need to pay the bills, if you’re setting up someone you like to do motivated work. But don’t play hurt if you can avoid it. Find something you love. Create something meaningful. Encourage an environment that allows people to work at their full potential. And take risks to do it.

The article ran in the Sunday paper and we were hoping folks would take it in to work and argue with coworkers about what numbers to assign to which band and we’ve been happy with how much traffic we saw. And a few hours ago a Metafilter story went up about it, which is pretty dang cool.

Today the first project I built launched, the Politics Glossary. It’s a small site cataloging political jargon and all the writing is done by the Politics department. Actually, one of the best things about this job is getting to meet folks at all levels and departments in the Post.

I build and maintain website projects that support stories from the paper or play to the unique strengths of the web as a medium. As examples, Faces of the Fallen can be seen as an expansion of obituaries and the U.S. Congress Votes Database is a large database that wouldn’t work in print. There are several dozen of these projects. The two I just linked are the two largest and most-frequently updated, though most are small and not constantly updated after their initial launch. This team is experimenting a lot with what it means to do journalism on the web and I’m proud to be part of it.

Obligatory Official Disclaimer: “My comments are entirely personal and do not represent the views of Washingtonpost.Newsweek Interactive or any other Washington Post Company affiliate.”

I’ll post about my new job when I have one. Thanks to my friends and family for all their support.

It’s a darn clever idea and I want to see them succeed. Go easy on the site’s rough edges, it’s truly beta, not that faux-beta that Google sticks finished products in for years to avoid doing support.

True to their CEO’s mail, I’m still user number 9. I’d link to my profile, but I can’t find a URL for *any* profiles. (Just one of those rough edges that true beta products have.) Contributors can earn cute little awards (think of Boy Scout badges) for sending in ideas and such. I’ve got a “9″ award with the text “Peter will always be #9 to us.”

I’ve poked around the site and even submitted an idea (I sniffed out the link for my idea, but it’s an uuugly URL and unfortunately doesn’t link to my profile). It’s like a fun project and I’m already a fan, so I’ll contribute to a project or two and blog about it.

I found out about it via Jon Wolter, who mentioned it at last night’s ChiPy meeting and at TechSocial.com, which is a nearly indispensible resource for Chicago geeks.

The puzzle is run by Cambrian House, a mysteeeeerious stealth startup that has only a teaser page online. If there’s one thing that annoys me more than Flash puzzle-level games, it’s stealth startups.

So I went poking around some more and found a development copy of their website. And much to my surprise, it’s actually a darn cool business idea. So cool that after I wrote a whole “Mwaha, I’m raising the curtain early!” post I thought better of it and am only going to post this for now. It was clever and fun, there’s going to be a ridiculous amount of buzz around this company when it launches. And I got to register as user #9, which was damn funny when I noticed their “About Us” page lists 17 employees.

Confidential to CH in Calgary: it’s really tacky to litter your URLs with “.php” and get variables. Put the following in your .htaccess file and have index.php take apart the URL with the PHP code below and route to your different pages. URLs like “/community/member/Harkins” look much nicer than “/community/member-profile.php?users_id=9″. (Or ask me about mod_rewrite.)

<Location /secret_development_environment>
    Order Deny,Allow
    Deny from all
    Allow from 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 .cambrianhouse.com
</Location>

RewriteEngine On
RewriteCond    %{REQUEST_FILENAME}  -d [OR]
RewriteCond    %{REQUEST_FILENAME}  -f
RewriteRule     ^(.*)$  - [L]

RewriteRule ^(index.*) - [QSA,L]
RewriteRule ^([^.]*)$ /index.php [QSA,L]

 
function url_parse() {
        $url = $_SERVER['REQUEST_URI'];
 
        // we don't want to pass get vars or anchor tags on to the script
        if (strpos($url, '?'))
                $url = substr($url, 0, strpos($url, '?'));
        if (strpos($url, '#'))
                $url = substr($url, 0, strpos($url, '#'));
 
        //remove leading slash and possible trailing slash, store in $url
        if (substr($url, 0, 1) == '/')
                $url = substr($url, 1);
        if (substr($url, -1) == '/')
                $url = substr($url, 0, -1);
        if ($url == '/')
                $url = '';
        $url = explode('/', $url);
        
        return($url);
}

You want a little healthy fear in your life, it keeps you from trying to pet those cute little bear cubs. In business, it keeps you paying attention to things like what the competition is up to, to if your burn rate is sustainable, and how important those last few bugs are.

Most NDAs exist because of two different and worthwhile fears: early competition and secret sauce. If you and two buddies have read Getting Real and struck out on your own to start a web-based business, you’d like to delay the day that knockoffs start appearing. Alternatively, if you’re Google, you have hordes of resourceful competitors and abusers who’d love to mine the offhand comments of your engineers.

But the nearly-as-common motivator behind NDAs is shame. You could call it fear of being found incompetent, but the word for that is shame. A shame-powered NDA will invariably be described as an important security measure, but the business is covering up that it runs everything in a slipshod, last-minute, “this is good for now and we need it” manner. Most organizations just barely work and spend their time lurching between crises, which is mildly disconcerting in an interdependent society but handy for breaking the spirit of idealistic young college graduates.

An NDA easily slips from being protection against competitors to being protection against customers, so companies have to be regularly act introverted, maybe stare into a candle while holding a crystal, and make sure they’re being honest with themselves about why they have an NDA.

So think about yours. If you’re not thinking about how to balance tipping your hand and bragging about how cool you are, something’s terribly wrong.

• Tags you’ve applied to yourself still appear in the tag cloud.
• Tags don’t always get marked with a number correctly.
• The numbers are 1, 2, 3 but the categories are Newbie, Skilled, Advanced.
• Clicking a tag adds a border to it, often moving it out from under your mouse.
• Clicking multiple times to move between boxes is n * lame.
• Drag and drop would have solved all of these.
• Never send or store cleartext passwords!
• Tags are global, so I can’t fix wrong case.
• Can’t use HTML in the “about me” blurb (“10,000 ft view”).
• Searching for candidates, the 1, 2, 3 are back. It’d work nicer with a little box you could click to switch between off, newbie, skilled, and advanced.

Last, the warning: you can have 64 tags (total between all categories). GoJobby doesn’t tell you this anywhere, you’ll just notice a tag randomly disappear every time you try to add a 65th. I noticed this because I started adding tags for all the fun odd stuff I’ve learned and then noticed things kept disappearing. You’ll have to sit there and count your tags to know how many you have, GoJobby is very young yet.

Despite my nitpicking and the couple small bugs, GoJobby is great to use and will pick up tons of job seekers. Unlike Dice and Monster, registration is easy and filling in a profile is dead easy. And you can actually upload your resume as a file — how 1995! It’s not just a little nicer for seekers, it’s worlds better.

Anyone else have suggestions or know of other web 2.0 job sites? I’m trying to think of others, and Indeed is the closest I can get.

Connect all four dots: draw three straight lines without lifting your pencil from the paper, and finish where you started. At first it looks impossible, but after your manager gets done chortling they’ll say your problem is that you need to “Think Outside the Box!”, show you the answer, and then go on to make tenous and tedious metaphors about creativity.

And that’s bullshit. An important part of creativity is working within constraints, such as the box pictured above, especially when they’re entirely arbitrary constraints on a meaningless project imposed by management fiat.

To truly solve this puzzle, we only need one straight line and the kind of mental agility that MBAs didn’t learn in a business marketing class. The correct answer looks like this:

You may, at this very moment, be thinking I am also bullshitting because that is so not a straight line. Your problem is that you need to Think Outside the Euclidean Universe!

When we look at life, it seems that it takes place in a flat 3d space where two parallel lines will go on being parallel for their entire length. But Einstein’s Theory of Relativity showed our universe is way less boring and predictable than that. Gravity bends spacetime, allowing cool things like gravitational lenses where light (normally travelling in a straight line from our perspective) passes by a massive object like a galaxy or a black hole and bends towards it. We don’t notice it as we sit in go about our lives because we don’t spend a lot of time very near black holes.

So, when you recognize the four dots appear in a non-Euclidean space, the answer is obvious. There’s a massive object warping spacetime between the four points, and because they’re only an inch or so apart it must be a black hole. If you were to represent the geometry of this distortion, the solution would look be even clearer:

So there we are: our manager’s problem is trivially solved when precipitously near to a black hole, so you’d better hope that the poor bastard your manager picks to implement this solution isn’t you.

Thanks to Eric Sandalle for the 3d render.

<mewyn> i’m gonna throttle one of our developers
<mewyn> we had to reboot a server because his crappy code leaked ALL the server’s memory, pys was 99% and swap was 99%
<SmooveB> heh
<Malaprop> mewyn: As long as you’re throttling devs, can I give you a list?
<Malaprop> Hm, I want someone to start a Web2.0 company, ThrottleYourDevs.com. Internet-enabled developer throttling, delivered right to your door.
<SmooveB> XmlHttpThrottleRequest.com
<Malaprop> I could add my list of throttled devs to my feed-reader to keep up with my order status.
<SmooveB> feeds, xmlhttprequest, mashups… perhaps some sort of google maps integration
<SmooveB> and realtime SVG tracking.
<Sketch> you can’t be web2.0 without google maps integration
<Malaprop> Totally. Beowulf clusters of throttled developers are so Web 1.0.

Note: No developers were actually throttled in the making of this post. But we wish some were.

Instead, write the most overblown ad copy you can imagine. Don’t just lavish praise on whatever you’re describing, lay it down with a firehose. After seeing enough commercials it’s dead easy to spew this crap, and it’s way more fun than lorem ipsum. Here’s an example:

Bill Lumberg, CEO of Initech, is the only person to ever have earned doctoral degrees from every American college. He bats .400 for the Chicago Cubs and dug the Panama Canal with a bent teaspoon while on sabbatical from neurosurgery. He can also fly and shoot laser beams from his eyes.

Your clients are not only unfamiliar with what lorem ipsum means, using it will confuse them and distract them from evaluating your design and code. Lorem ipsum can just go die for several reasons. This reason’s just good fun, and it’s not like it hurts to have clients who’re smiling while they evaluate your work.

If you hate this idea and think your clients would too, you’re probably wearing a tie.