Washington Post Update
Code: , , ,
15 comments

It’s been a long time since I’ve written
about what I’ve been up to at the Washington Post, so let me run down the apps
I’ve worked on since September 2007 in roughly linear order. Lots of these apps link different places,
so if you don’t see projects.washingtonpost.com at the start of the
URL, you’ve probably wandered off something I directly worked on.

Campaign Tracker
I made it possible to browse events by type. I’ve also gotten data to academic researchers who’ve requested copies of this (I think) unique data.
Voters Guide
The 2006-8 Metro Voters Guide is probably the clunkiest app I worked on. Lots of confusion between producing a one-off site and a maintained app, as well as between primaries and generals. 2009 will be better because we’ve settled those issues and we’ve built a nice database schema for dealing with elections as part of the 2008 Presidential race.
Congress Votes
I loaded the new info for the current and previous sessions of Congress, added the ability to view just current members of Congress (as some join or leave during the term), added JSON feeds that power the Apture displays (yes, I know about the NYT Congress API, I’m getting the signoffs to publicize the machine-readable pages).
Staff E-Mail Form
Mostly just maintenance, but some generalization for shared mailboxes.
Challenge Index
Formerly the clunkiest app, we fixed it by refocusing it on local schools (dropping national) and I rebuilt it in Dec 2008 to clean up the code. Then we imported data back to 1996, and it’s a really nice little app. Also the source of my 3rd rule of database apps (watch for the post).
Presidential Endorsements
During the primaries, we tracked endorsements from public figures, mostly politicians. The app is interesting today: the front page only mentions the last few candidates like Clinton, Obama, and McCain because we stopped advertising the candidates as they dropped, but their data is all still in the db. We’ve been meaning to resurface them now that the election is settled, but it’s hard to remember to find the time to be an archive instead of just news.
Sellout Songs
This went along with a cute piece in the Sunday Style section. Fun fact: Python has one constant for pi, the graphic used another, and the example songs used a third. If you play with the app, you’ll see it matches the examples (because that’s the only discrepancy folks might notice). This project is where I started dreading the Sunday paper: because there’s a Saturday edition (the “bulldog” edition), things have to be ready late Friday evening. And print folks are under a daily pressure that has driven them all a bit mad, so they tend to finish projects right up against the deadline. Since this project we’ve managed projects better to avoid the “OK, now that it’s 7 PM on Friday, here’s everything you need to build the web app” problem, and I’ve noticed the web has gotten more consideration in the print newsroom over time.
Campaign Finance Search
Candidates are required to report contributions over $200 to the FEC and we pull that data from them. We were showing state, city, and zip code totals, then I built a search engine to expose the individual records. (And we also track noteworthy donors.) Take a good look at the search page and results table so you know it as you continue reading this list.
Dog Parks
A small app showing off Dog Parks in the DC Metro area. We’ve talked about adding more features, but it’s never climbed high enough up the priorities ladder.
Recipes of the Day widget
The only (technically) interesting thing about this project was preloading a month’s worth of season-appropriate recipes at a time, because it’d grind down a producer to have to update this by hand every day. It’s a nice toy.
Silent Injustice
The FBI used to analyze the content of lead in bullets to match a bullet in a victim to bullets owned by a defendant. Trouble is, the science didn’t really work, and they quietly dropped it without notifying people convicted in whole or in part on this unreliable evidence. I helped put all the case files we could find online. This was a really rewarding project: it may have only mattered to a few hundred defendants and their loves ones, but it meant a lot to them.
Primary Pages
They showed live news and election results on election nights, and our database server struggled a little to keep up that first Iowa night. Postgresql doesn’t have good replication options like MySQL, so we beefed up the hardware before Super Tuesday and have been running great since. On some of the election nights, traffic to the projects server accounted for more than 10% of all WashingtonPost.com traffic.
Primary Exit Polls
Lots of fun breakdowns of data by demographic, location, response, etc. One thing I noticed on this project is that it’s hard for humans to talk about breaking down data in more than three dimensions (and three itself can be tricky). Funny how our visual and spatial skills affect the rest of our cognition.
Post 200
The Post 200 showcases 200 prominent local businesses; apparently it’s been a yearly staple for a few decades. In 2008 we built a site we could continuously update, rather than just drop a little data once a year. I’d still like to grow this app to encompass all local companies, or all publicly traded companies (because more data makes everything better, right?). The Post doesn’t do a whole lot of business coverage, though, so we may not be able to add enough value to make that all worthwhile.
Clinton’s Schedule
In the 2008 primaries, Hillary Clinton supported her claim of being more experienced with her time as First Lady, and eventually acquiesced to the call to release her schedule so people could judge her time. I found out the day before (while chatting at the water cooler) that her campaign would be dropping it off at the National Archives. Seemed like it might be fun to do something with the data, but we had no idea what format it would take. The next morning a courier picked up a CD from the Archives and we saw it in all its glory: a 17,481 page PDF that looked like it had been scanned from the fax of a photcopied inkjet printout. If you look at the news stories written that day (2008-03-19) you’ll notice they all take the form of “What was Clinton doing on this important date?” because reporters couldn’t search, just page through it chronologically. I used 200 Amazon EC2 instances to break up and OCR the entire PDF in a few hours and we put the searchable database online (and it became an AWS Case Study (note: PR people wrote that thing)). The project was an absolute blast: making interesting data available by applying cutting-edge tools on a tight deadline.
Young Lives At Risk
I pulled nutritional data on 11k foods from the USDA (in tilde-delimited format, weird) and it powered an interactive shopping cart. Over the few weeks of this project, the USDA called me nearly daily to chat about nutritional data. I respect their commitment to nutritional data, but by the end of it I just wanted to buy them all puppies so my phone would stop ringing.
Sacred Ground
I didn’t work on the (very nice) Flash app, I updated the Faces of the Fallen app to user-submitted tributes and the “Sacred Ground” templates for Pentagon victims. This looks really nice if you’re surfing from this story on the opening of the Pentagon Memorial, but it’s hugely confusing if you’re visiting from the main Fallen app. It’s one of those last-minute compromises that sometimes happen, and I plan to repair the damage.
Business Glossary
When the CDO market finally had to face reality and cratered, the business dept. asked if it would be possible to build a resource for all the (unfortunately) new terms readers were confused by. Working from the Politics Glossary (gee, eerily similar), we were able to respond with an entire application instead of just a promise to build one.
Thrift Savings Plan data
The iframe near the bottom of the page with TSP market information (sort of a 401k for government employees) is a small app that pulls data from the TSP.
Vote Monitor
Our team kept some empty space in our schedule because we knew the Politics team would surely come up with a last-minute idea for an election app, and they did. After some debate about whether to stay local or open up to national reports, we built a great app to allow people to report their voting experiences. I wish we’d had the idea sooner to better advertise it and work closer with other organizations that monitored the elections. Note for 2010 to-do list…
Obama Transition Donations
Hey, remember that table with donations presidential candidates table from back when? Obama, big on transparency, has released information on the donations to his presidential transition and we’re presenting a fairly familiar view of it. I believe they’ll release the data on January donors in about two weeks, so we’ll be updating then.
Clinton Foundation Donations
Hey, remember that table with donations… yeah, you remember. As part of Hillary Clinton’s confirmation process as Secretary of State, the Obama team required the Clinton Foundation to release previously-private donation information. They grudgingly posted it to their website, which crashed under the load. We scraped what we could as the site was intermittently up and presented it.
Bush Legacy Timeline
Finally, reaching a project that went live two days ago, the Bush Legacy Timeline graphs key indicators against important events. I created the database to store events and helped out with the Flash coding.

Some other projects that I haven’t done much more than bugfix are Local Blog Directory, Recipes, DC Schools, Political Ads, Faces of the Fallen, High School Recruiting, and Inauguration Alerts. I’ve also built a few internal tools to automate or streamline the editorial process, but they don’t make for compelling reading and this post is already plenty long.

In case you’ve wondered why I’ve called the Post the best job I’ve ever had, scroll back up and read it again. Tons of interesting, meaningful projects with more talented professionals than I could manage to list, but let me try so this post doesn’t make me look like a braggart:

The foundation was laid by Adrian Holovaty and Derek Willis. It’s the nicest codebase I’ve ever inherited. Almost every one of the apps I listed is well-designed because of Alyson Hurt. Several of these data sets came from Sarah Cohen. I’ve worked a bit with the developers Dan Berko and Ed Holzinger. Ryan O’Neil is the other developer on the Tools Team who builds all these fun apps with me, and he occasionally saves me from my bugs and design decisions. There’s also all all the news producers who created and managed apps: Jason Manning, Sarah Lovenheim, Paul Volpe, Andrea Caumont, Amy Kovac, Tanya Ballard, Amanda McGrath, Jacqueline Dupree, Alicia Cypress, and Amanda Zamora. Last, it’s been possible because of great bosses Bob Greiner, Karl Eisenhower, Jim Brady, and Dave Baker.

More TK.

Don’t Play Hurt
Code: , , ,
5 comments

I love the c2 wiki. At least once a year I’ll dive in for a day or two and read a swath through its accumulated wisdom. As I learn and experience colors my perspective, I always find new things.

One of the most important articles I’ve read on the c2 wiki wasn’t about a programming technique. It was PlayHurt. The core debate of the page is whether software developers can produce good code even when their hearts are not in it. Because development is a creative mental activity, it’s profoundly affected by the developer’s mental state. When I first read the page, one comment on it really caught my attention:

Is it possible that few of us have ever done anything but programming for a company while playing hurt? Companies seem to put most programmers into unproductive situations most of the time. Without properly supporting a technical developer, then developers PlayHurt and probably fail (the 95% of software projects fail rule again). For example if the marketing is not in place and we don’t have and never will have any real customers then we are playing hurt no matter how good a job we do on the code.

I was working on a product that was unrelated to the rest of the company’s business. It had originally been built in a hurry by inexperienced developers in an unfamiliar language, and no one inside or outside the company had a complete understanding of the product (self included). The CEO set the goal to increase sales 10,000% percent by the year’s end without creating a marketing budget (10% growth would have been lucky). The company culture was built around churning recent college graduates, so there were low standards and no professional criticism.

I don’t often read things and think they speak exactly to my situation, but that comment sure did. It put a name to what I was feeling. I felt terrible because I didn’t understand why I didn’t want to go to work in the mornings and why I had to push myself to fix every bug and complete every feature. I didn’t have the perspective or experience to recognize all the things I just listed that had gone wrong, so I felt like a failure because I couldn’t develop on command. I found some of that understanding on the wiki. I couldn’t Play Hurt anymore. I was just plain tired of being hurt. I quit that month.

If you want to write good code, you can’t play hurt. If you want to be proud of your work, you can’t play hurt. If you want to feel you’re making something of your life, you can’t play hurt. The best work is powered by passion and meaning, not obligation. Play hurt if you’re getting through the occasional frustrations that every job has, if you need to pay the bills, if you’re setting up someone you like to do motivated work. But don’t play hurt if you can avoid it. Find something you love. Create something meaningful. Encourage an environment that allows people to work at their full potential. And take risks to do it.

Post: The Moby Quotient
Biz: , ,
3 comments

On Friday I put up an app to let folks calculate and share the Moby Quotient of songs — a measurement of how much an artist sold out by allowing their work to be used in a commercial. The app is a companion to a story in the Post.

The article ran in the Sunday paper and we were hoping folks would take it in to work and argue with coworkers about what numbers to assign to which band and we’ve been happy with how much traffic we saw. And a few hours ago a Metafilter story went up about it, which is pretty dang cool.

Post: Politics Glossary
Biz: , , ,
2 comments

Politics Glossary logo

Today the first project I built launched, the Politics Glossary. It’s a small site cataloging political jargon and all the writing is done by the Politics department. Actually, one of the best things about this job is getting to meet folks at all levels and departments in the Post.

WashingtonPost.com
Biz: , ,
2 comments

Today I added code to the Washington Post’s 2007 Presidential Field to show campaign events by type. I’ve joined Washington Post/Newsweek Interactive as a web developer (Python with Django) and this is my first code to go live.

I build and maintain website projects that support stories from the paper or play to the unique strengths of the web as a medium. As examples, Faces of the Fallen can be seen as an expansion of obituaries and the U.S. Congress Votes Database is a large database that wouldn’t work in print. There are several dozen of these projects. The two I just linked are the two largest and most-frequently updated, though most are small and not constantly updated after their initial launch. This team is experimenting a lot
with what it means to do journalism on the web and I’m proud to be part of it.

Obligatory Official Disclaimer: “My comments are entirely personal and do not represent the views of Washingtonpost.Newsweek Interactive or any other Washington Post Company affiliate.”

Good Timing
Code: , , , ,
2 comments

This morning, at about 4:30 AM, I awoke and just knew the Right Way to rebuild RegistryPro to be completely reliable, even more compact, and provide meaningful reporting. It would take less than two weeks of coding time and the pitfalls are well-demarcated and avoidable. It would be really great if I’d thought of it two years ago when I still worked there. Thanks, brain.

Leaving Hostway
Biz:
No comments

Earlier today I gave my two weeks’ notice to Hostway. I’ve worked at Hostway since April 2005 as a maintenence coder on RegistryPro and, since last month, part-time on the web frontend of an internal monitoring system. I’m not going to be posting about why. There’s no exciting scandal, it’s just a case of mismatched needs and goals.

I’ll post about my new job when I have one. Thanks to my friends and family for all their support.

Cambrian House Party
Biz: , ,
2 comments

Cambrian House default avatar
Cambrian House, the startup I snuck a peek at and got a hat-tip from has opened for a public beta test. They’re sort of an open-source business incubator: folks submit ideas, the best of which become projects; folks submit code, art, and copy, the best of which go into the finished project. Cambrian House (or a spinoff company, perhaps) runs the project as an online business, paying royalties back to the folks who contributed.

It’s a darn clever idea and I want to see them succeed. Go easy on the site’s rough edges, it’s truly beta, not that faux-beta that Google sticks finished products in for years to avoid doing support.

Cambrian House "9" Award
True to their CEO’s mail, I’m still user number 9. I’d link to my profile, but I can’t find a URL for *any* profiles. (Just one of those rough edges that true beta products have.) Contributors can earn cute little awards (think of Boy Scout badges) for sending in ideas and such. I’ve got a “9” award with the text “Peter will always be #9 to us.”

I’ve poked around the site and even submitted an idea (I sniffed out the link for my idea, but it’s an uuugly URL and unfortunately doesn’t link to my profile). It’s like a fun project and I’m already a fan, so I’ll contribute to a project or two and blog about it.

TECHcocktail
Biz: ,
1 comment

Despite its odd capitalization, I’ll be attending TECHcocktail, a technology/business mixer on July 6, 2006.

I found out about it via Jon Wolter, who mentioned it at last night’s ChiPy meeting and at TechSocial.com, which is a nearly indispensible resource for Chicago geeks.

Pre-Cambrian House
Biz: , , , , ,
6 comments

I was poking around reddit and followed a link to CambrianCode.com, an all-Flash (ugh!) puzzle game. There’s a few of these “guess how to get to next level” games online and they all just annoy me. Yes, you’re so clever. No, I find patronization alluring. Yes, I’ll spend my time on this for no discernable reward.

The puzzle is run by Cambrian House, a mysteeeeerious stealth startup that has only a teaser page online. If there’s one thing that annoys me more than Flash puzzle-level games, it’s stealth startups.

So I went poking around some more and found a development copy of their website. And much to my surprise, it’s actually a darn cool business idea. So cool that after I wrote a whole “Mwaha, I’m raising the curtain early!” post I thought better of it and am only going to post this for now. It was clever and fun, there’s going to be a ridiculous amount of buzz around this company when it launches. And I got to register as user #9, which was damn funny when I noticed their “About Us” page lists 17 employees.

Confidential to CH in Calgary: it’s really tacky to litter your URLs with “.php” and get variables. Put the following in your .htaccess file and have index.php take apart the URL with the PHP code below and route to your different pages. URLs like “/community/member/Harkins” look much nicer than “/community/member-profile.php?users_id=9”. (Or ask me about mod_rewrite.)

<Location /secret_development_environment>
    Order Deny,Allow
    Deny from all
    Allow from 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 .cambrianhouse.com
</Location>

RewriteEngine On
RewriteCond    %{REQUEST_FILENAME}  -d [OR]
RewriteCond    %{REQUEST_FILENAME}  -f
RewriteRule     ^(.*)$  - [L]

RewriteRule ^(index.*) - [QSA,L]
RewriteRule ^([^.]*)$ /index.php [QSA,L]
function url_parse() {
        $url = $_SERVER['REQUEST_URI'];
 
        // we don't want to pass get vars or anchor tags on to the script
        if (strpos($url, '?'))
                $url = substr($url, 0, strpos($url, '?'));
        if (strpos($url, '#'))
                $url = substr($url, 0, strpos($url, '#'));
 
        //remove leading slash and possible trailing slash, store in $url
        if (substr($url, 0, 1) == '/')
                $url = substr($url, 1);
        if (substr($url, -1) == '/')
                $url = substr($url, 0, -1);
        if ($url == '/')
                $url = '';
        $url = explode('/', $url);
 
        return($url);
}
Cambrian House house responded to me.

Fatal error: Call to undefined function twentyseventeen_get_svg() in /home/malaprop/push.cx/wp-content/themes/pushcx/archive.php on line 45