Why is my bug not fixed ? why is the UI still so unpleasant ? why is performance still poor ? why does it consume more memory than necessary ? why is it getting slower to start ? why ? why ? – the answer lies with developers: Will you help us make OpenOffice.org better ?

Meeks identifies the lack of growth in the community as the core problem and his suggestions for improving the software product are about redesigning the community to be more responsive to developers, to welcome newbies better, and to simplify the act of contributing. Notably, he doesn’t suggest a single technical change for improving bugs, the UI, performance, or other features.

Maybe this is a better example than mine of Django and Rails: developers don’t tend to get irrationally competitive about office productivity suites and stop listening like they do web frameworks. (This is part of why OOo is in more trouble than Django.) The point is the same, though: a healthy, active community is the only guarantee of a an open source project’s viability. Community determines whether a project keeps up with its competition and redefines the limits of its field, or maintains the status quo, or becomes unreliable and unusable.

Communities need to be active to continue improving the project, deal with bugs and changes to their ecosystem. If no one is interested enough to talk about the project, none of that will happen. Newcomers need to meet experienced users to be sold on why to use the software, to get help as they learn their way around, to maybe be drawn into contributing to the project itself.

Some communities are maladjusted. They self-defeat (see the section on Bion’s support groups), they lack plans or goals, they abuse newbies to keep from growing or changing, they’re more interested in bringing something down or defending turf than building up their own project. Some successful projects start with a grudge to pick, but most that do so fail to develop their own identify, and you can hear that in a cacophony of blame and flames in their social spaces.

Productivity is the best way to tell if a community is well. Even if it doesn’t have regular releases, it needs explicit plans on a reasonable schedule to meet measurable goals. Good software is produced by careful and deliberate iteration.

All well and good, but what about the slugfest between Django and Rails? When do we get to the part where I reveal myself as an irrational, deceptive propagandist because I say anything nice about the project you don’t prefer? Better question: why is there so much vitrol between the two?

The two are far more alike than dissimilar: extracted from projects that were created in similar dynamic languages from late 2003 through 2004, their core functionality was routing URLs to templates rendered with data retrieved with an ORM.

The biggest early difference is in these two contemporaneous blog posts: Django releases, Rails turns 1. Django was wrongly seen as a Python copy of Rails on its release, though it’s just a case of convergent evolution: to solve similar problems they developed along the same lines.

If success is binary, both Rails and Django are successes. They’re strong, reliable tools for building websites quickly. They are their respective languages’ leading web frameworks. Speaking personally, I enjoy working daily in both.

If success is relative, Rails is far more successful than Django. By any measure I can take, the Rails community is at least ten times larger. For example, Indeed has more jobs and Google Trends shows more searches:

And yesterday I posted about the difference in books with a ridiculous title. The difference between eight books and dozens can’t be explained away by Python being simpler, the Django docs being better, or Rails being more of a moving target (all true); none of them would stop publishers from trying to profit from an audience. The jobs, the searches, the books: they’re imprecise metrics that all point the same direction.

Rewind. Rails and Django were roughly technologically equivalent. They were both once a little flaky and incomplete and Definitely Didn’t Scale. But with the aid of good marketing, Rails turned its earlier release into a lead in in the developer market and created a vibrant, growing community (even if that community acts out a self-parody roughly annually).

Rails has had more resources in the form of bloggers, coders, writers, trainers, and users, and it’s used them to push in far more directions than Django has. Rails has explored dead ends like components, built plugins and changed course to use better integration with its package manager, and soon the entire Merb sort-of-fork project is merging into Rails with the results of its successful experiments.

Django hasn’t moved nearly as fast, it hasn’t had the resources to make as many mistakes to learn from. The Django community is recovering from an 18 month gap between feature releases (0.96 on March 23, 2007 and 1.0 on September 3, 2008). 1.0 included a lot of great features and improvements, but it was a long, quiet year-and-half getting there while Rails was continuing to iterate.

It’s not a question comparing 1.0 minus .96 against 2.2 minus 1.2.3. It’s about activity and momentum, Django lost it while Rails has used the network effect to grow the most active web development community to build better software. Count the amount of blog posts, consultancies, newbies, rants, silliness, projects, and weaknesses addressed instead of books and see how Rails has succeeded. It is its own ecosystem instead of only being part of the web.

The difference in relative success is that Django’s still a useful tool for building good websites. Meanwhile, Rails has Changed The Web, influencing graphic design, pushing collaboration, proving the value of Javascript toolkits, driving the adoption of distributed version control. I’m not claiming Rails is singlehandedly responsible for these trends, I’m saying it’s been a dominant force in these advancements of the web.

The Django and Rails communities fight so much and so rudely because they’re too similar not to. Projects survive and thrive because they have attention from developers, and developers don’t often pick up two tools to do the same job (usually it’s one tool for every job). Community is the lifeblood of every open source project and the world of difference between Django and Rails is evidence.

Rails

In Conclusion

The difference speaks for itself. I’ll write about why this matters tomorrow.

(Yeah, I used the Fancy Link Builder I posted yesterday about. It was infuriating to use, it has whiz-bang Javascript to open a product detail lightbox that served only to make it nearly impossible to copy cover images. New theory: the creators of the Fancy Link Builder are moles from offline bookstores who have infiltrated Amazon to destroy it from within.)

Campaign Tracker

I made it possible to browse events by type. I’ve also gotten data to academic researchers who’ve requested copies of this (I think) unique data.

Voters Guide

The 2006-8 Metro Voters Guide is probably the clunkiest app I worked on. Lots of confusion between producing a one-off site and a maintained app, as well as between primaries and generals. 2009 will be better because we’ve settled those issues and we’ve built a nice database schema for dealing with elections as part of the 2008 Presidential race.

Congress Votes

I loaded the new info for the current and previous sessions of Congress, added the ability to view just current members of Congress (as some join or leave during the term), added JSON feeds that power the Apture displays (yes, I know about the NYT Congress API, I’m getting the signoffs to publicize the machine-readable pages).

Staff E-Mail Form

Mostly just maintenance, but some generalization for shared mailboxes.

Challenge Index

Formerly the clunkiest app, we fixed it by refocusing it on local schools (dropping national) and I rebuilt it in Dec 2008 to clean up the code. Then we imported data back to 1996, and it’s a really nice little app. Also the source of my 3rd rule of database apps (watch for the post).

Presidential Endorsements

During the primaries, we tracked endorsements from public figures, mostly politicians. The app is interesting today: the front page only mentions the last few candidates like Clinton, Obama, and McCain because we stopped advertising the candidates as they dropped, but their data is all still in the db. We’ve been meaning to resurface them now that the election is settled, but it’s hard to remember to find the time to be an archive instead of just news.

Sellout Songs

This went along with a cute piece in the Sunday Style section. Fun fact: Python has one constant for pi, the graphic used another, and the example songs used a third. If you play with the app, you’ll see it matches the examples (because that’s the only discrepancy folks might notice). This project is where I started dreading the Sunday paper: because there’s a Saturday edition (the “bulldog” edition), things have to be ready late Friday evening. And print folks are under a daily pressure that has driven them all a bit mad, so they tend to finish projects right up against the deadline. Since this project we’ve managed projects better to avoid the “OK, now that it’s 7 PM on Friday, here’s everything you need to build the web app” problem, and I’ve noticed the web has gotten more consideration in the print newsroom over time.

Campaign Finance Search

Candidates are required to report contributions over $200 to the FEC and we pull that data from them. We were showing state, city, and zip code totals, then I built a search engine to expose the individual records. (And we also track noteworthy donors.) Take a good look at the search page and results table so you know it as you continue reading this list.

Dog Parks

A small app showing off Dog Parks in the DC Metro area. We’ve talked about adding more features, but it’s never climbed high enough up the priorities ladder.

Recipes of the Day widget

The only (technically) interesting thing about this project was preloading a month’s worth of season-appropriate recipes at a time, because it’d grind down a producer to have to update this by hand every day. It’s a nice toy.

Silent Injustice

The FBI used to analyze the content of lead in bullets to match a bullet in a victim to bullets owned by a defendant. Trouble is, the science didn’t really work, and they quietly dropped it without notifying people convicted in whole or in part on this unreliable evidence. I helped put all the case files we could find online. This was a really rewarding project: it may have only mattered to a few hundred defendants and their loves ones, but it meant a lot to them.

Primary Pages

They showed live news and election results on election nights, and our database server struggled a little to keep up that first Iowa night. Postgresql doesn’t have good replication options like MySQL, so we beefed up the hardware before Super Tuesday and have been running great since. On some of the election nights, traffic to the projects server accounted for more than 10% of all WashingtonPost.com traffic.

Primary Exit Polls

Lots of fun breakdowns of data by demographic, location, response, etc. One thing I noticed on this project is that it’s hard for humans to talk about breaking down data in more than three dimensions (and three itself can be tricky). Funny how our visual and spatial skills affect the rest of our cognition.

Post 200

The Post 200 showcases 200 prominent local businesses; apparently it’s been a yearly staple for a few decades. In 2008 we built a site we could continuously update, rather than just drop a little data once a year. I’d still like to grow this app to encompass all local companies, or all publicly traded companies (because more data makes everything better, right?). The Post doesn’t do a whole lot of business coverage, though, so we may not be able to add enough value to make that all worthwhile.

Clinton’s Schedule

In the 2008 primaries, Hillary Clinton supported her claim of being more experienced with her time as First Lady, and eventually acquiesced to the call to release her schedule so people could judge her time. I found out the day before (while chatting at the water cooler) that her campaign would be dropping it off at the National Archives. Seemed like it might be fun to do something with the data, but we had no idea what format it would take. The next morning a courier picked up a CD from the Archives and we saw it in all its glory: a 17,481 page PDF that looked like it had been scanned from the fax of a photcopied inkjet printout. If you look at the news stories written that day (2008-03-19) you’ll notice they all take the form of “What was Clinton doing on this important date?” because reporters couldn’t search, just page through it chronologically. I used 200 Amazon EC2 instances to break up and OCR the entire PDF in a few hours and we put the searchable database online (and it became an AWS Case Study (note: PR people wrote that thing)). The project was an absolute blast: making interesting data available by applying cutting-edge tools on a tight deadline.

Young Lives At Risk

I pulled nutritional data on 11k foods from the USDA (in tilde-delimited format, weird) and it powered an interactive shopping cart. Over the few weeks of this project, the USDA called me nearly daily to chat about nutritional data. I respect their commitment to nutritional data, but by the end of it I just wanted to buy them all puppies so my phone would stop ringing.

Sacred Ground

I didn’t work on the (very nice) Flash app, I updated the Faces of the Fallen app to user-submitted tributes and the “Sacred Ground” templates for Pentagon victims. This looks really nice if you’re surfing from this story on the opening of the Pentagon Memorial, but it’s hugely confusing if you’re visiting from the main Fallen app. It’s one of those last-minute compromises that sometimes happen, and I plan to repair the damage.

Business Glossary

When the CDO market finally had to face reality and cratered, the business dept. asked if it would be possible to build a resource for all the (unfortunately) new terms readers were confused by. Working from the Politics Glossary (gee, eerily similar), we were able to respond with an entire application instead of just a promise to build one.

Thrift Savings Plan data

The iframe near the bottom of the page with TSP market information (sort of a 401k for government employees) is a small app that pulls data from the TSP.

Vote Monitor

Our team kept some empty space in our schedule because we knew the Politics team would surely come up with a last-minute idea for an election app, and they did. After some debate about whether to stay local or open up to national reports, we built a great app to allow people to report their voting experiences. I wish we’d had the idea sooner to better advertise it and work closer with other organizations that monitored the elections. Note for 2010 to-do list…

Obama Transition Donations

Hey, remember that table with donations presidential candidates table from back when? Obama, big on transparency, has released information on the donations to his presidential transition and we’re presenting a fairly familiar view of it. I believe they’ll release the data on January donors in about two weeks, so we’ll be updating then.

Clinton Foundation Donations

Hey, remember that table with donations… yeah, you remember. As part of Hillary Clinton’s confirmation process as Secretary of State, the Obama team required the Clinton Foundation to release previously-private donation information. They grudgingly posted it to their website, which crashed under the load. We scraped what we could as the site was intermittently up and presented it.

Bush Legacy Timeline

Finally, reaching a project that went live two days ago, the Bush Legacy Timeline graphs key indicators against important events. I created the database to store events and helped out with the Flash coding.

Some other projects that I haven’t done much more than bugfix are Local Blog Directory, Recipes, DC Schools, Political Ads, Faces of the Fallen, High School Recruiting, and Inauguration Alerts. I’ve also built a few internal tools to automate or streamline the editorial process, but they don’t make for compelling reading and this post is already plenty long.

In case you’ve wondered why I’ve called the Post the best job I’ve ever had, scroll back up and read it again. Tons of interesting, meaningful projects with more talented professionals than I could manage to list, but let me try so this post doesn’t make me look like a braggart:

The foundation was laid by Adrian Holovaty and Derek Willis. It’s the nicest codebase I’ve ever inherited. Almost every one of the apps I listed is well-designed because of Alyson Hurt. Several of these data sets came from Sarah Cohen. I’ve worked a bit with the developers Dan Berko and Ed Holzinger. Ryan O’Neil is the other developer on the Tools Team who builds all these fun apps with me, and he occasionally saves me from my bugs and design decisions. There’s also all all the news producers who created and managed apps: Jason Manning, Sarah Lovenheim, Paul Volpe, Andrea Caumont, Amy Kovac, Tanya Ballard, Amanda McGrath, Jacqueline Dupree, Alicia Cypress, and Amanda Zamora. Last, it’s been possible because of great bosses Bob Greiner, Karl Eisenhower, Jim Brady, and Dave Baker.

More TK.

The standard workaround is to loop over the list and zip the hash data into some kind of bigger list, but this is expensive if the list is big and just plain annoying, especially if you’re on Django 0.96 and can’t unpack tuples with your for tag.

So I finally hacked up a template tag to give me access to dictionary contents:

 
@register.filter
def hash(h, key):
    return h[key]

And it’s used like:

 
{% for o in objects %}
  <li>{{ dictionary|hash:o.id }}</li>
{% endfor %}

It may not be beautiful, but it mostly works. Annoyingly, you can’t do dictionary|hash:o.id|hash:'foo' to get at an inner hash — anyone know why?

I suspect it’s because the admin interface also defines custom tags in a file named log.py and that you should avoid duplicating any template tags filename in your INSTALLED_APPS.

This public service announcement has been brought to you by a wasted hour of my life.

 
from django.db import transaction
 
@transaction.commit_manually
def view(request):
    # you make changes to your objects, calling obj.save() on them as normal
 
    # when you commit, all changes to all objects will be commited to your database
    transaction.commit() 
 
    # an excellent idea would be to wrap in a try/except block
    try:
        # do your stuff
    except:
        transaction.rollback()
    else:
        transaction.commit()

I couldn’t find a good doc quick for Python decorators (that bit with the @ before the function declaration) for people who haven’t used them before, so if someone has a good one, please post it in the comments below.

Here’s a link to the full docs and for those interested in the nitty-gritty, here’s a link to the code checkin.

And before you ask, magic-removal is expected to be merged to trunk before the end of March.

Actually, let me admit up front that five years is a shot in the dark, and optimistic to boot. If people keep hanging out with bondage and discipline languages like Java and C# that are still catching up to language and framework developments from the 90’s it’ll take us more like ten years. (Attention Lisp Weenies: Yes, I know you solved every problem forty years ago for certain values of “solved” and “problem” while the rest of us were getting work done.) Not only is versioning a difficult technical problem, it will be difficult to educate programmers in what it is, how it works, and why you’ll wish you used about a year after you decided it was too much work.

I’m writing to get help get the ball rolling on the process of solving this problem and publicizing it. So maybe it’s about time I go into what the problem is instead of just yammering on, eh?

Because your website is served from your machines, you can update it every few days or hours. This is almost pedantically obvious, but it’s a big change from having to press CDs, ship them to a store, and wait for them to go home with customers — it’s even a big change from posting a new version online for customers to download and install.

The web’s faster pace means that updating, say, your database model to add or remove fields is a common occurence rather than something your InstallShield wizard does every 1-2 years. It’s vital that changes be streamlined and safeguarded. What kind of defaults get set when you add fields? How do you save data when you lose fields? How does the programmer make sure the Right Thing happens? If you have the fields Name and Address and want to combine them into a simple MailingAddress field, you’re not just deleting two columns and adding one.

Not only is it complex to change your schema, getting those changes into production are not easy. The script that updates definitions and migrates data (which we tested somewhere other than production, right?) needs to placate the multiple web and database servers used for load-balancing. If you update the database server first, the application code on the web server shouldn’t break because it didn’t get the word about the new schema.

I see too many problems with trying to get multiple servers to update to a new version at the Exact Same Instant, I think schema changes are going to have to be a four-step process with a backwards-compatible first step applied to the database (add columns, loosen restrictions), a code update (with testing!), a backwards-incompatible database change (drop columns, tighten restrictions), and finally another code update to remove outdated usage.

Jakob Kaplan-Moss, one of the creators of Django, just opened the discussion of versioning in Django (with links to prior art). As a web framework with an object-relational mapper, versioning is an important feature.

In addition to databases, web APIs need versioning. Adam Kelsey has opened that discussion, but so far the solutions are pretty rough and I don’t have anything to add to the discussion.

If you’d like to read more about how web development differs from other development, Steve Yegge wrote an excellent article titled It’s Not Software coining the term “servware” to emphasize that web developers need to recognize the ways our code ends up so different. (It’s easy to turn this topic into rah-rah “We’re so special we don’t have to learn from history” bullshit, but nobody wants that.)