I’m processing all the items in a big list, and part of that is performing some expensive calculation on an item attribute. There are only a few dozen values for the attribute, so I cache them:

 
results = {}
 
for item in really_big_list:
    if item.attribute in results:
        result = results[item.attribute]
    else:
        result = expensive_calculation(item.attribute)
    # ... do something useful with item and result

It’s simple and readable, but it’s also lengthy. If I’m doing several expensive calculations on different attributes (and I am), my actual work gets lost in the noise. So I defined a dictionary that can do the heavy operation and cache the result:

 
class LambdaDict(dict):
    def __init__(self, l):                                                                                                                       
        super(LambdaDict, self).__init__()
        self.l = l
 
    def __getitem__(self, key):                                                                                                                  
        if key in self:
            return self.get(key)
        else:
            self.__setitem__(key, self.l(key))
            return self.get(key)
 
# and now my code becomes
 
results = LambdaDict(lambda  key:expensive_calculation(key))
 
for item in really_big_list:
    result = results[item.attribute]
    # ... do something useful with item and result
 

That’s really nice, clean code that I’m satisfied with. The difference in Python and Ruby here is that Ruby hashes (the equivalent of dicts) include this behavior by default, just pass a block (lambda) when constructing the hash and it’ll be called for every missing key.

I’ll have the same behavior in Python or Ruby, it’s just that the default Python object doesn’t give me a handy method for building a dictionary that might perform some arbitrary expensive method on a simple lookup. Ruby is in favor of implicit magic, so it holds out the hook. This is the difference I was trying to get at: Python’s builtin objects have fewer methods and convenient hooks for me to do weird and useful things than Ruby’s, and Ruby will even let me tinker with them. Ruby has open classes, so I can extend both the builtin and defined classes with my own methods:

 
module Enumerable # included by Arrays and similar objects
  def sum
    inject(0) { |x, y| x + y }
  end
end

This adds a sum() method every Array in my program, whether I construct it (like I created my own results dict) or get it from some library code. In Python I’d have to define my own Array type and I’d be out of luck if I’m getting back arrays from a library. Monkey patching is Python’s (deliberately clunky) name for adding a method to a single object at runtime. I’d have to monkey patch every object as it’s returned from the library rather than just being able to declare that every object of that class should have my method.

If this is the first time you’ve seen open classes: yes, when I first saw it I felt exactly the same way you do. Dangerous unreliable hackery, a recipe for disaster. But I’ve seen a lot of useful things happen in Rails because of it, and my Ruby projects have benefited from being able to add a method to an existing library’s objects, to overwrite (or just chain my code before or after) another method.

It’s something of a last resort that lets me build the cleanest object system I can as I interface with builtin objects and library code. I don’t have any utility methods floating around or have to tweak each object as I get it back from an API call. As I cook, Ruby’s metaprogramming is the garnish that finishes the dish. I called Python academically inconvenient because it feels like there’s an academic designer at my shoulder saying, “No, you don’t want to do that, it’d be messy and might be be abused. Pedagogically unsound.” I love that Python pushes me to build an explicit and obvious code, but I find myself wanting to tuck in one little bit of magic to make the code perfect.

 
s = s[:255][:max(s.rindex('.'), s.rindex('!'), s.rindex('?')) + 1]

This snippet chops it down to the 255 max, finds the ., !, or ? marking the end of the last sentence, and chops there. Great, right? Except it doesn’t work.

Instead of returning None when it can’t match the substring, rindex throws ValueError. So unless the first 255 characters of the string contain a ., !, and ? it’ll throw an exception. OK, let’s try:

 
rightmost = -1
try:
    rightmost = s.rindex('.')
except ValueError:
    pass
try:
    rightmost = max(rightmost, s.rindex('!'))
except ValueError:
    pass
try:
    rightmost = max(rightmost, s.rindex('?'))
except ValueError:
    pass
s = s[:255][:rightmost + 1]

Eww. OK, let’s encapsulate that redundancy:

 
def no_exception_rindex(s, substring):
    try:
        return s.rindex(substring)
    except ValueError:
        return None
 
s = s[:255][:max(no_exception_rindex(s, '.'), no_exception_rindex(s, '!'), no_exception_rindex(s, '?')) + 1]

That’s… well, it’s at least a little better. Lucky that max doesn’t mind seeing None, I could imagine it throwing its own ValueError. But I wouldn’t call this code good, we’ve been forced to switch out of object-oriented code because we can’t add our no_exception_rindex to the string objects.

Here’s another approach:

 
def rightmost_punctuation(s):
    index = len(s) - 1
    while index > 0 and s[index] not in ['.', '!', '?']:
        index -= 1
    return index
 
s = s[:255][:rightmost_punctuation(s) + 1]

I’d actually call this one worse, as it’s not immediately obvious what it’s doing. And anytime I create a variable and then tinker with it inside a loop I feel like I want to rewrite that code to use map and/or reduce.

Tomorrow I’ll redo this example in Ruby to talk about open classes, but for today does anyone have a better approach in Python?

Feihong Hsu presented lessons learned on doing Unicode in Python. I love hearing real-world experience, and Feihong knows his stuff. After his presentation there was a discussion where other folks chipped in their notes and answered questions. The gist of it is that Python is good for Unicode, and in Python 3000 it’ll be great.

Ian Bicking is working part time on the One Laptop Per Child project and brought a prototype. I couldn’t get over how nice it is, it has a lot of thoughtful design touches like the antennae doubling as latches to close it. I’ve heard a lot of folks say they want one, but you really don’t — the keyboard is physically too small for adults to type on. Ian mentioned it’s currently fairly difficult to get started developing, but there’s a lot of attention on the problem and he’d like to set up a sprint to develop for the laptop after dev envs are easy to set up.

Ted Pollari took a few minutes to talk about how he and others worked to bring PyCon to Chicago in 2008. They’re finalizing the hotel contract this week, and planning for a thousand developers. Ted thanked everyone who helped, and encourages anyone who’s curious or might be able to volunteer to join the pycon-organizers mailing list.

Pete Fein presented Grassy Knoll a REST-style interface to PyLucene using JSON as the data format on the wire. REST is a natural fit for the document management and searching, so it looks like a good project. He’s looking for folks to lend a hand.

So to talk about the office: it’s a nice, if very deliberate space. It’s nice to see that someone thought about all the little decorative touches (like lava lamps) and useful touches (like an acoustic shell above where the projector displayed), but it didn’t feel very lived-in yet. Maybe it was just that having to sign an NDA to enter the office for a public meeting was an unpleasant surprise.

First up, Brian Ray gave a presentation on operator overloading in Python. He covered what operators are available, how Python picks which operand’s code is called, how assignment operators like += and -= are handled, and tips on good style.

Next, Brian Fitzpatrick and Ben Collins-Sussman talked about how they’ve been working on Google Code, specifically the free Project Hosting for open source software. The highlight (and clearly Fitz’s favorite part of the project) was how they used Google’s BigTable as their svn repository.

Last, Jason Huggins demo’d Selenium, the browser-based website testing framework he developed. He had a nice demo showing how Selenium can be integrated into a continuous integration system so that browser tests are run on multiple OSs on every checkin, with screen-capture movies of the test process automatically checked into a repository.

All-in-all, this was a very fun meeting with lots of conversation that didn’t break up until security wanted to close up at 23:00 (5h after the meeting started). I had fun meeting and chatting with at least a half-dozen new folks. Mostly I yammered on and on about how Cambrian House works, but I swear it’s because folks kept asking and not because I’m turning myself into a mobile billboard.

Also, Jonathan Wolter talked me into coming out to San Francisco with him and a few others to attend Yahoo’s Hack Day in two weeks. Looks to be a good time.

Unicode puts out PDF code charts of each and every glyph so people can have some idea of what they should look like (though the actual display to the user is left up to fonts). If you’d like to see one, the chart for Linear B is a nice little example. Because I didn’t want to mess around with putting dozens of PDFs together, I just grabbed the single 30MB 777-page PDF for the latest version, 4.1.0. (New versions are released as they add more scripts — cuneiform is scheduled for 5.0, yay!)

Looking at the PDF (whether you grabbed the small example or decided to be macho and pulled up the whole thing), it’s obvious that the “one big table per page” approach makes it pretty easy to extract the glyphs. I wrote a program to do this, because it would be stupid and ultimately self-defeating to open the PDF with GIMP to manually select and save each glyph.

First, I used ghostscript to convert the PDF to a series of 600 DPI pngs, one image per page. This took about 39 minutes and 471MB of disk space. (I’ll explain why I converted at such a high resolution in about two paragraphs.)

Next, I wrote a Python script using the Python Imaging Library to analyze every page, locate the glyph table if the page has one, and walk through the table exporting each glyph to its own individual PNG. The 769MB of individual glyph images took about 5.6 hours to rip. And that’s OK! I kept the code simple to read because performance doesn’t matter. That’s an odd thing to say for a program that takes hours to finish, but you only ever run this program once. Note, that’s never an excuse for shoddy quality, because you never run a program just once. Yes, those two statements are at odds. If you don’t see that they’re both true, you have not yet learned the Tao of Code.

Well, there’s one important step elided from the previous paragraph: figuring out which glyph is which so I know what the filename should be. Every glyph is labeled with a four or five-digit hexadecimal number, but of course the label is part of the image rather than text. Converting images to text is known as OCR (optical character recognition) and it’s difficult to avoid errors. In my favor I had images that were big (600 DPI, see, I told you I’d get back to it), error-free, and black-and-white; but a strike against me was that it can’t spell-check a word like “A29C”. After trying a few different OCR programs, I got ocrad working well (with a little tinkering, it often mistook the number 0 for the letter O).

Well, mission accomplished. I’ve ripped 97,715 glyphs from the Unicode code charts to individual PNGs. So what should I do with them all?

I work at Hostway, a web hosting company with about a dozen offices worldwide. I picked up a copy of our logo in vector format from one of our graphic designers and resized it until it had about the same number of black pixels as I have glyphs.

Then, of course, I wrote another script to render a final image. It created a huge new image in which every pixel from the source was a downsampled (because I’m printing at 300 DPI) glyph, and it tries to make sure it puts the darkest glyphs in place of the darkest pixel so I have a (very) rough antialiasing effect to soften the edges of letters. (If it’s not already painfully obvious, yes, I did ascii art as a teenager.) This script immediately crashed and burned.

The problem is scale. The source image is 1618 x 151 pixels, and I’m outputting at 300 DPI: 1618 x 151 x 300 x 300 = 20.9GB. As cool as it would be, I just don’t have that much RAM. I had to break up the image into pieces three feet wide, which is good because that’s also how wide the biggest printer in the office is. And I got to write a great error message: “Can’t create images this big. Reduce max_piece_width or buy more RAM.”

Made of 16 images totaling 310MB (it took 45m), the Hostway logo worked out to be 6 feet tall and 46 feet long. My boss told me it was “too big” to hang up. I told him that was “exactly the point of the whole thing”. I’ll change his mind yet — everyone in the office I show it to loves it, so it’s just a matter of time.

I presented the code at the June 2006 meeting of ChiPy and it was well-received. One attendee told me afterwards it was the funniest, nerdiest presentation he’s ever seen, which I consider high praise. As promised, I’m sharing the source code.

unicode-art.2006-06-15.tar.gz

You’ll have to be a little familiar with graphics and Python to make use of this. It is not a copy of all the glyph graphics, and I will not provide you with them or make posters for you, so please don’t ask. I’d love to hear what about what you do with it, please do something cool and post a comment or trackback.

I know a few folks are dropping by looking for the source I promised to post. It’ll be up later tonight or tomorrow morning — I had to recreate one of my scripts from scratch and I’d like to make sure I did it right so I don’t post broken code. And as all of you at the presentation understand, running any of these scripts takes a few hours. So it’ll be up Real Soon Now.

Thanks again to Fawad Halim for loaning me the use of his laptop for my presentation.

Of course it took a little longer than I expected, but it’s up now.

Source: Pretty much everyone the first time they hear of DSLs.

Answer: No, a DSL is much more than a library, and I have an example that won’t make you say, “Well, sure, if you’re doing something that esoteric…”

My example of a domain-specific language is strings. No, seriously. Let’s figure out the length of a string in x86 assembly:

 
strlen:
	push	edi
	sub	ecx, ecx
	mov	edi, [esp+8]
	not	ecx
	sub	al, al
	cld
repne	scasb
	not	ecx
	pop	edi
	lea	eax, [ecx-1]
	ret

Computer memory is one big linear stream of bytes we can scan through, looking for the null that terminates our string. Boy, is that some fast code — we might even call it efficient, if we ignore the fact that we’ll reach the eventual heat death of the universe before we finish our web 2.0 app.

So we move up in abstraction to C, which has arrays. And you can pretend a string is an array and walk it looking for that null terminator:

 
static int my_strlen(const char *c) {
    int l = 0;
    while (*c++) l++;
    return l;
}

This code is basically the same as in assembly, but it must be nicer to read because it uses all that stylish punctuation. Well, it’s not really nicer, maybe a string isn’t really just an array. So let’s look at Python:

 
len("Hello, world!")

Now that’s downright human-readable. And I’ll admit I’m fudging here by just calling the built-in len() instead of writing one, but it just works and there’s none of this messing around with null bytes.

Well, maybe there’s messing around with null bytes. I don’t have to know how Python implements len() and, more importantly, I don’t have to pretend a string is only an array or a small bit of sequentially-addressed memory.

To continue the example let’s look at regular expressions, a powerful way to search strings. We’ll write a pirate detector in Python:

 
import re
matches = re.search("ar+g+h", "Oim a poirate, arrrgh!")
if matches:
    print "There must be a pirate, I heard someone say '%s'." % matches.group()
else:
    print "No pirates detected."

This is important code, as pirates hide all over the web. But it’s pretty clunky, we have to import a library and call functions and evaluating responses and save objects… It’d sure be handy if regular expressions were part of the language like in Ruby:

 
if "Oim a poirate, arrrgh!" =~ /ar+g+h/ then
  puts "There must be a pirate, I heard someone say '#{$&}'."
else
  puts "No pirates detected."
end

This code is even nicer, our regexp is a first-class type and tightly integrated into the language. The increase in stylish puncutation might make for a higher learning curve, but we can express ourselves much more naturally.

A DSL is all about moving up in abstraction until your code directly reflects the high-level concepts you’re working in. You don’t have to peer through a haze of bits and pointers, your actions become synonymous with your intentions.

A lambda expression defines an anonymous function. Here’s a regular function definition:

 
def inc(x):
    return x + 1

This definition binds the name inc in the local namespace to a function object. To get the exact same functionality using lambda, assign the lambda expression to a variable:

 
inc = lambda x: x + 1

A lambda has two parts: the argument list (only one arg in this case) before the: and an expression after. It can’t contain statements (like assignments or print) because it’s an expression itself. This is a pretty useless example, so let me show how I used it today.

When you call list.sort(), Python sorts it by calling the built-in cmp() function on pairs of elements from the list, which is defined as:

 
def cmp(x, y):
    if (x < y):
        return -1
    elif (x > y):
        return 1
    else
        return 0

My problem was that I was sorting a list of lists, a couple rows of statistics for a spreadsheet. Python was sorting it by the columns in order from left to right, and I needed to sort by one of the middle columns. Luckily, you can pass a new cmp function object to sort().

 
list.sort(lambda x,y: cmp(x[2], y[2]))

The lambda defines a function that takes arguments x, y and calls the built-in cmp function with item two of the inner list. The sort happens an inside column of the list rather than the first.

I could have defined a function, but that would be overkill for code I’d only call from one place. This simpler and more readable. It’s not a big win, but every little bit helps.

As a side note, Python 2.4 added a key argument to sort() that simplifies my code. If it’s set, Python calls it with the list element and passes what it returns into cmp. I figured the cmp example would show off lambda better, but in case you’re curious:

 
list.sort(key=lambda x:x[1])

It’s been tough to write this and not start pulling in concepts from functional programming. Maybe I’ll get to use them next week. In the meantime, I’d like to hear more real-world uses of lambda if you’ve got one to share.

Let’s look at the final version of the Card class and its usage again:

 
values = ('2', '3', '4', '5', '6', '7', '8', '9', '10', 'J', 'Q', 'K', 'A')
suits = ('h', 'c', 'd', 's')
 
class Card:
    def __init__(self, value, suit):
        self.value, self.suit = value, suit
 
    def __repr__(self):
        return "<Card: %s%s>" % (self.value, self.card)
 
    def __eq__(self, card):
        return self.value == card.value and self.suit == card.suit
 
    def __ne__(self, card):
       return not self.__eq__(card)
 
>>> c1 = Card('J', 'h')
>>> c2 = Card('J', 'h')
>>> c1 == c2
True
>>> id(c1), id(c2)
(-1210559028, -1210558804)

As designed in the previous blog post, two different objects (as seen by their different ids) equate because of the custom __eq__() handler.

But this is a bit silly — Cards are so small and simple, having identical objects is wasteful. If we’re writing the code for a casino, we don’t really want several thousand Jacks of Hearts, we want many references to one object. The idea is that instantiating a Card object checks a hidden pool of objects for a duplicate and returns it, creating a new object only if needed. (Code based on that of Duncan Booth.)

 
import weakref
 
class Card(object):
    _CardPool = weakref.WeakValueDictionary()
 
    def __new__(cls, value, suit):
        obj = Card._CardPool.get(value + suit, None)
        if not obj:
            obj = object.__new__(cls)
            Card._CardPool[value + suit] = obj
 
        return obj
 
    def __init__(self, value, suit):
        self.value, self.suit = value, suit

Notice that this uses new-style classes. (Side note: I cringe at anything named “new”, “next”, “updated”, etc. because they tell me nothing. If the only thing you can say about a reimplementation is that it’s not the old one, put down your copy of Refactoring and back away from the computer. Take up writing toothpaste advertisements.) If you’re wondering what’s up with __new__, Guido explains the difference best:

__new__ is the first step in instance construction, invoked before __init__. The __new__ method is called with the class as its first argument; its responsibility is to return a new instance of that class. Compare this to __init__: __init__ is called with an instance as its first argument, and it doesn’t return anything; its responsibility is to initialize the instance.

The other possibly-unfamiliar element is the WeakValueDictionary. In a nutshell, it’s a dictionary that deletes its entries if no other variables point to them. When instantiating a Card object, it checks in the Card class’s _CardPool to return an existing object, or creates a new Card object if needed.

If you’re confused, take a few minutes to play with the code. Insert print statements into __new__ to show you the contents of _CardPool.items() and whether a new object is created. There’s a lot of important object orientation concepts at work in a short block of code: inheritance, the difference between classes and objects, references, and overriding.

I’ve removed the __eq__ and __ne__ calls because two Cards with the same value and suit are the same objects, saving our casino many gigs of precious RAM:

 
>>> c1 = Card('9', 'h')
>>> c2 = Card('9', 'h')
>>> c1 == c2
True
>>> id(c1), id(c2)
(-1210940772, -1210940772)

The c2 wiki has more info on the Flyweight Pattern.

Here’s the basic Card class. Note that I’m going to totally skip things like error-checking and documentation to keep the example obvious.

 
values = ('2', '3', '4', '5', '6', '7', '8', '9', '10', 'J', 'Q', 'K', 'A')
suits = ('h', 'c', 'd', 's')
 
class Card:
    def __init__(self, value, suit):
        self.value, self.suit = value, suit
 
    def __repr__(self):
        return "<Card: %s%s>" % (self.value, self.card)

Man, does code get short when you don’t bother checking for errors. The usage is pretty clear, but there’s one odd issue:

 
>>> Card('3', 's')
<Card: 3s>
>>> Card('3', 's') == Card('3', 's')
False

Huh? That’s odd, an instance of Card doesn’t equal another Card just like itself? Well, let’s look at the Python docs. It talks a bit about comparing the builtin types (numbers, strings, lists…) and then says: “Most other types compare unequal unless they are the same object”.

Python does this by comparing the ids of the objects. You can call id() on your objects and see that even identically-constructed objects have different ids because they’re in different locations in memory. This is decent default because you wouldn’t want Python walking deeply through all of your objects, a potentially expensive operation. Python does do a little bit more for equality, as implied by that “most” in the documentation.

Python does one more thing for us. It looks for a function named __eq__ to call on the left-hand object and uses it to determine equality. So let’s add it to Card:

 
    def __eq__(self, card):
        return self.value == card.value and self.suit == card.suit

Easy enough. And usage:

 
>>> Card('3', 's') == Card('3', 's')
True
>>> Card('3', 's') == Card('K', 'h')
False
>>> Card('3', 's') != Card('3', 's')
True

Now that last one is a bit surprising. Back at the docs, we learn “There are no implied relationships among the comparison operators. The truth of x==y does not imply that x!=y is false. Accordingly, when defining __eq__(), one should also define __ne__() so that the operators will behave as expected.” That’s:

 
    def __ne__(self, card):
        return not self.__eq__(card)

After that, the Cards equate properly and everything’s happy. One of the best things about Python is that it regularly gives you a sensible default and then lets you customize your code to work seamlessly with the language. This is what us Python fans mean when we go on and on about code being “Pythonic”.