Developer Day Notes


Category: Code
Tags:  IRC

I’m attending Developer Day DC and taking notes. They’re fairly rough and sort of only cover the things that catch my attention, but here they are in the hopes that other people find them useful:

Developer Day Notes 1: Opening Keynote

by Jay Virdy, speakerrate Nice visualization of ratings… instead of showing an average of reviews, it shows a gradient of worst-to-best, a horizontal stacked bar chart. Started from Amazon, do text analysis to figure out if they were for or against. Realized they weren’t getting any attention from search engines, so they decided to aggregate content. Then they started indexing blogs and rating products, examples: Iron Man movie and Audition: A Memoir (book). So they figured out topic, then aggregated reviews and comments. Had a nice bit of launch traffic from a TechCrunch article and Google indexing, but Google booted them out of results along with PayPerPost. Switched to indexing Twitter in 3 weeks as they were running out of money and their investors gave up on them. Launched Apr 11 2008. 50-100% traffic growth week over week; acquired by Twitter in July. Takeaways: Solve problems for innovators/early adopters in your target market. Strive for good enough, not perfection. Build something simple and the market will pull you in. Q&A: * Real-time, scaling, and trending were the hard parts. Plugging in Twitter was easy. * 6 people in the company. * Have to be prepared to capitalize on opportunities.

Developer Day Notes 2: Give Your Sites a Push with Comet

by Dan Drinkard Goals: What is it, how does it, how do I, the future? In short, a method to push to browser instead of client pull (eg. polling). Polling suffers from latency, event chunking, unnecessary bandwidth use, and connection overhead. Comet work will touch the entire stack; browser, server, message broker/protocol/models. Bayeux is sort of synonymous with Comet for many people. It’s a Dojo Foundation project for solving common Comet problems, basically a set of protocols (content encoding, formatting, API, no guarantees, etc.). Some folks use XMPP, nice and robust and stable. Stomp, Streaming Text-Oriented Messaging Protocol has some fans. Looks sort of like HTTP. Message Brokers take info from messaging layer to app, Apache ActiveMQ (v popular), RabbitMQ (Java/.Net/C#), MorbidQ (good for testing). Transports – last mile to browser: Long-Polling: Simplest, just hold open connection and then respond. Client retries on timeout. Looks like only good for one event. “3x max latency” – worst case if you just miss an event is: Response, Request, Response. Forever Frame: iframe points to a chunked resource that flushes occasionally. Fast. XHR Streaming: Same thing but with XHR, fire OnreadyStateChange but [maybe] only WebKit supports it. ActiveX: htmlfile, IE-only so… booooo. Server-Sent Events: HTML 5 spec so of course Opera only. Maybe cool in future. Sockets: Straight TCP via plugin like Flash, Silverlight. Optimal speed but plugins may choke on lots of data. Problems: 2-connection limit, same-origin policy, still half-duplex because HTTP fundamentally is. Example: what are Twitter users listening to now? Server: Orbited, built on Py/Twisted with Ruby and PHP APIs. Easy to set up, socket-like interface. Real easy install and short config. [Code looks sort of Rails-like in that it gives you places to drop your poller, display code, etc. and you fill them in. Nice.] Trouble with eating RAM and connections, and possible you’ll have proxy/firewall troubles. Real trouble is just that the community is young and fragmented and the whole thing is sort of a hack. Q&A: Security is especially young, mostly it’s your app’s problem. Vendors promise 10k-100k idle connections per server, but big grain of salt.

Developer Day Notes 3: Natural Language Acceptance Testing

by Patrick Reagan Small shop, QA is a part-time job of developer and project manager. But quality is important and testing communicates the intent of code… and maybe communicate user expectations of the code. Goals: developer & PM pair, tests first, drive a web browser. Start with a story: User can register for an account (success state). Requires email, pwd, pwd confirm, accept TOS (the form). Acceptance includes no dupe emails, pwd at least 6 chars (error conditions). Behavior is defined by locations (URL), page state (existence of elements), and message content (eg flash response). Should define state, steps to execute, and expectation of results. Manual testing is straightforward but time-consuming, error-prone, and most importantly isn’t a regression suite so you don’t know if you’re improving or breaking things. Selenium IDE is a macro recorder for your browser. Easy to use, but the HTML table output is sort of weird. And it’s easy to have problems with browser state, like if you run login a second time it’ll fail because you’re already logged in and because a user by that name is registered. It’s hard to maintain, reuse, or understand. Selenium Remote Control lets you write in a familiar programming language. Requires a proxy Java server to drive traffic. Mostly you can do black-box testing, but you’ll probably need a URL to create a clean state. But it still requires a developer and is not readable by non-developers. Can the test statement be executable? State: * Given no users exist in the system Steps:

  1. Visit the registration page
  2. And I enter valid registration information
  3. And I submit the form Expectation:
  4. Then I should be redirected to the home page
  5. And I should see a success message displayed Yes. Cucumber is a Ruby testing tool that speaks to customers/domain experts first and code second. Create a Feature that has a readable English statement of what the steps and expectations. When you first run Cucumber, it’ll create the skeleton so you can define the verbs and nouns. Given /no users exist in the system/ { open ‘clean_slate_url’ } Problems: Still depends on a developer and no user control over input data. But you can make this dynamic, change the steps to say things like “When I enter ‘user@host.com’ to the ‘user_email’ field” Cucumber will create a skeleton that picks up those values and your tests can use them. What’s important is that this makes it extensible — a method to enter text into a text form field can be reused on later tests. Then you can move higher up, make a template that says “And I enter ‘’ for ‘user_email’” and create a table of email/pwd combos and it’ll run the test for each row of data. (Demo of the example test in the slides) In the future, you can refactor tests to remove duplication and data like “Given a user exists with the email ‘’” or ‘And I enter valid registration data’. One problem with this testing is that it’s fairly slow, on the order of 10s for a single test. But you can run specific named tests or tag tests to be able to run a subset easily. Selenium RC is easy to run cross-platform, you can make it run a browser in a VM so you can test browser/OS combos with continuous integration. Q&A: How do you deal with scale? Tests could take 40m. A: Uh, no really good answer to this, unfortunately. What’s your experience with it? A: It can cut down on back-and-forth between customers and devs. But I haven’t used it on a production project.

Developer Day Notes 4: A Ruby 1.9 Case Study on Upgrading RCov

aaronbedra.com github.com/abredra Ruby is young and maturing, Ruby 1.9 will be a major improvement (especially in speed). You’ll need to maintain separate environments for Ruby 1.8 and Ruby 1.9 to be able to start playing. Demo of shell scripts to user_ruby_191 or user_jruby. One-command switch of the PATH, GEM_HOME, and GEM_PATH environment variables. RCov tells you C0 code coverage of your test suite. To know if things will work in Ruby 1.9, just pick the project and run its test. It will almost certainly break. RCov’s C extension was its claim to fame, made it 200-300x faster than pure Ruby tools. But reaching into Ruby internals means the upgrade to 1.9 was especially hairy. Remove of node.h and env.h is/was a big problem as rcov relied heavily on them. The easy parts are that len and ptr are no longer members of array and string, but there are new macros… but backwards compatibility is now a pain in the ass, so you’ll have to create a wrapping macro, or separate code branches and let mkmf do the work. RCov looks like an old gem. Maybe it could use Jeweler, but right now it needs to work with 1.9. So, back to the Ruby side. First, small things: Use ‘then’ instead of : in case statements. String no longer has to_a (use lines.to_a) or several other shortcuts. The main RCov binary is basically a big option parser. ‘gem install rcov’ is the abandoned Rubyforge, but ‘gem install relevance-rcov’ from GitHub is the current version. lowlevel.rb is a Ruby-only proof of concept so that they know code coverage will work and they’re using it as a guide for developing the C extension. C extensions will be your timesink. Most of the Ruby changes are very straightforward. “rexml is no longer your friend” – it’s less flexible and hard to make backwards-compatible libs, but much faster. Q&A: zsh scripts at github.com/spicycode/spicy-config

Developer Day Notes 5: Lightning Talks

Impromptu – Andrew Bedra

Scheme, hooks into OS X: ports, camera, goal is creating art in Lisp. Code live to create art in front of an audience Audio track, built-in iSight, and an OpenGL presentation that will go along with it. Starting with a piano sample; create a base melody loop. Create an OpenGL “presence” and define an animation… in this case, a rolling spiral and tweaks to constants are saved each time he evals the buffer. Adding to the structure of the music, the animation. “The idea here is that you can start from a simple cube an animate it, rotate it, turn it into a sphere, end up with all kinds of spheres.” He can put in video, capture images or the stream, add filters like bloom, blur, blend. “You can imagine a concert shaping up with this.” ### Comet Experience – Mohan Sukara Virtual team, have been working together about 3 years. Most of us are in Europe. When they lost a project it prompted them to introspect on process and they decided to build a project management tool that pulled together their chat, project tracker, and task to-do list. Started with TCL but moved to Rails. Real-time was a puzzle, though. Someoe heard of Comet and decided to build in Python where good tools were available. So we built even before we knew exactly where we were going and we built something nice. We sent it to TechCrunch, entered the TechCrunch 50, and it’s live at samepage.in. Thinks it’s built in Comet, Twisted and Dojo but he’s not the techie. One problem was that they started building a whole platform and they raised funding that… disappeared with the crunch. But they saw opportunities to create products, like real-time stock quotes to trading firms that needed to get over a 6-8s delay. You don’t know where things will take you, so just embrace it, make your mistakes, and go on. Apps for America 2 – Jeremy Carbaugh A bit of “reverse shameless self-promotion”. He works at the Sunlight Foundation [which rules] and wanted to talk about the launch of data.gov. Over the next month they’ll go from about 40 to maybe 8,000 data sets. So Sunlight is having a contest, if you create an open source app based on data.gov app there are significant cash prizes and a free trip to the Gov 2.0 Expo “This is your democracy, take ahold of it.” ### Progressive Caching – Ben Scofield This is part of his previous Developer Day talk on this topic, sort of a continuation. Rails has three caches: page caching (entire HTML output), action caching (runs normal filters but otherwise skips work), fragment caching (bits of HTML pages). MyPullList.com is an app for comic book fans who want to track the comics they wish to buy each week. Standard approach is to grab all the items, loop over them, and show the “pull” badge on the comic if the user wants to pull it. A better approach is to cache the HTML so it can come straight off the disk, but then how do you add the pull badges that differ by user? Well, when the page is loaded, you can use AJAX to make a second request that returns JSON with the individual user’s data to add the pull badges. Instead of 617ms for a response from the server, it’s about 135ms for the first request + AJAX request. Wow! A great way to cache the common public stuff and send just the custom stuff. This is such a win because the initial request bypasses all the Rails stack. On many pages (examples are Odeo.com and Amazon.com), usually large parts of the page can be cached except a few personalized bits. You can’t cache Squidoo pages (personal “you’d be interested in” results) and Twitter (personal subscribe list), but it’s nice. One problem, though, is that this depends on Javascript. (Hulu’s implementation is… unusual, at best.) Q: Why not just use the fragment caching and also cache the ? A: You’re still hitting the Rails stack. It’s a great technique, but that’s sometimes too slow for some applications. Q: Another technique is ESI. A: Yes, ESI works nicely. It might even let you replace this. Q: ESI? A: Edge Side Includes. Created by Akamai and caches fragments in Akamai or Varnish.

Developer Day Notes 6: Browser-Based Visual Programming with Lily and the Monome

by Justin Marney Goals: See amusic generator built using a Visutal Programming Language. Learn when, how, and why to use a VPL. Use the VPL Lily to mash the DOM, JS, and the Monome together. Party. The Monome is a physical computing interface with open-source hardware, hand-made. [damn cool] First, a demo based on Conway’s Game of Life. (Other VPLs: Max MSP, Pure Data). Idea! Use the Game of Life to generate MIDI. Create the start state, evolve every n ms, interpret state as a series of MIDI notes, display on the Monome, and reset to the start state every x iterations. Then, make it multi-track. (Awesome demo of light and msuic commences.) It involves setting patterns on the Monome and tweaking volume/samples on the monitor in Open Sound Control, a VPL. Starts simple and grows quickly complex as the patterns evolve. And the msuic is quite listenable techo/glitch. Thisis a “fringe technology”. It allows you to quickly create a working application with a usable interface, but you probably won’t be able to apply any of it to your day job. Toolbox: Monome (optional), VPL Lily (like PureData but in the browser), and “A community of smart people and open-source software”. The Monome is totally programmable. It’s a box of buttons, lighting up is decoupled: pusing a button sends a signal, your code could light it up instantly or not as your app decides. Apps exist for sound/midi generation, pixel animators, games, application control surfaces, video controls. “It’s minimal, but not so minimal you can’t get something done with it… these constraints fuel your creativity.” VPL programs are like big state machines. Boxes implement functionality and arrows connect boxes to pass data. No syntax to learn, often highly intuitive, and programs can be manipulated while they are running. Mostly they’re used by people doing live art (music/video/performance). (A demo of Lily.) It installs as a Firefox Add-On. Shows a small bit of code, a “patch”, that creates an array of names and maps them through string interpolation and print to print “Hello, X”. Because it’s in your browser, you can make the browser open a URL, analyze the DOM, even alter the DOM. Demo: open Craigslist, shuffle all the

tags and write them back to the DOM. “So this is crazy. Why would you ever do anything like this? Well, you wouldn’t.” This demo is not especially useful, but you could pull in Javascript libraries or scripts, maybe set it up so that choosing a word opens a lightbox filled with related pictures from Flickr. You can even save patches to create a standalone Firefox Add-On. (Shows js code to create Lily boxes.) Looks very clean and straightforward. You can use HTML do display things (in this case a checkbox form input) in your Lily patch. Open Sound Control is a network communication protocol that uses URL-style naming. “Who’s heard of OSC? One guy? Yeah, that guy uses his computer to make music.” MIDI is nice, but showing its age and can be slow. OSC is a replacement. Monome OSC protocol sends simple “/name/led x y 1” to turn on or “…0” to turn off a light, similarly simple for keypresses. “Weird mashup” demo: turn the DOM into a series of sounds so you could “play” a web page. Example page: startrek.com. Resources: lillyapp.org monome.org github.com/gotascii/sonifier github.com/gotascii/thunderdome cycling74.com puredata.info Q&A: Q: There’s monome support for Impromptu. A: There’s monome support for anything open. Q: I’ve had problems with copy and paste in code. How do VPLs deal with that? A: You can create js classes and reuse those, and create a box that contains a patch to pull it in many times. Overall, it feels a lot like functional programming.

Developer Day Notes 7: JavaScript Testing in Rails

by Larry Karnowski It’s TAFT, not TATFT* (*except JavaScript). Want to test a simple js app. Let’s test with Blue Ridge, first by installing the plugin to the example app (app is avail. on GH). It has some basic js, looks a lot like RSpec or Micronaut. Doing “rake test:javascripts” runs the tests on the command line and “rake js:fixtures” to get a project.html file he can open a browser and run tests from. If you install the TextMate bundle and hit Apple-R, it’ll open a TextMate window and run the tests there. The app comes with some basic functionality: a form for a Project (which has a js class), you can add rows to the form, and it has a stub reply of “Ask again later”. So in Blue Ridge, let’s create the javascript spec for the Project js class: “script/generate javascript_spec project”. The first run of this skeleton fails because the main project js uses the jQuery defaultValue plugin. So he stubs it out in the spec_helper.js and his specs run. He writes a test for a function to pull all the languages out of the example form, then turns to create a fixture for the test. The simple way is to edit the test HTML file and create the form elements. It works. But when he creates another test for the empty set when there are no languages, that file method gets clunky fast. So he uses a fixture() method using jQuery DOM creation methods in his spec to define the fixture, as well as setup/teardown code to instantiate fixtures and clear them afterwards. With a quick Rakefile edit, it’s easy to make the default “rake” task run the JavaScript tests after the usual Ruby tests. So what are all the moving parts? First, ScrewUnit is a js port of RSpec. Rhino is the Java implementation of JavaScript so you can run it in the JVM from the command-line. The env.js is a fake implementation of the DOM that is loaded for ScrewUnit. You can do every sort of normal DOM manipulation at the CLI. We use Run Code Run, an app we run at Relevance for continuous integration. There’s a nice Blue Ridge fork of the ScrewUnit TextMade bundle on karnowski’s GitHub account. Next, testing and coding the response to the form. There’s a js mock library called Smoke bundled with Blue Ridge that makes this testing straightforward, feels a lot like mocha. He works very step-by-step “do the simplest possible thing”, very XP style where code will do things like “return ‘no’;” instead of doing actual work to drive the creation of good tests. Suggests doing “ping pong” pair development where one person writes tests and the other person writes implementation. “Has anyone here does Test-Driven Development of JavaScript? (No hands.) Well, now you can.” Q&A: Q: Putting fixtures in code seems like redundancy. Can you remove it? A: Yeah, it’s sort of redundant, but you have to have it because the browser and cli work differently. Q: (Something about js standards.) A: The JS community is not test-infected, outside of jQuery. I’d like to see that change, and Blue Ridge go from Rails plugin to gem so it can be used more places.

Developer Day Notes 8: Making Life More Enjoyable With Python

by Jeremy Carbaugh Basic intro to Python. REPL (like irb), scripts vs. modules, objects, sys module, sdist, PyPI (Python package dir), setuptools (based on distutils for install), pip (like gem/apt), virtualenv (virtual environments for your different Python projects). virtualenvwrapper (Bash scripts for common virtualenv tasks). (No details here because I’m familiar with Python and felt like saving on the typing. Also it was kind of dry, I think because the presenter was subbing in for a coworker.) Final set of commands to get someone started at Sunlight Labs is: $ mkvirtualenv labs (labs)$ svn co http:///.…/trunk/ sunlightlabs (labs)$ cd sunlightlbags (labs)$ pip install -r reqs.txt (labs)$ mate . (labs)$ ./manage.py runserver Q&A: Q: How would you compare to Ruby? A: Very similar but I don’t have the hands-on experience to compare and constrast. Q: How do Python Eggs relate? A: Eggs are a very weird archive of your code, basically a zip file. But there are some really weird things, so more people avoid eggs.

Developer Day Notes 9: Programming In Interesting Times

by Russ Olsen “I think we’re really living in interesting times in terms of technology, in terms of programming.” Back at the start of his career in 1980 (“Dawn of Time”) every programmer was in one of two parts of Hell: the only options were Cobol (the west side) or, what he did, Fortran (the east side). “There was no lower-case in Fortran, so you were pretty much always screaming at yourself.” He read an article guessing what programming in the 21st century would look like (eg. where we are now) and the author admitted to not knowing what sort of language would be used or what the problems would be, but claimed that programming language would be called Fortran. “We were never going to get rid of Fortran, ever.” Scarily, Cobol and Fortran were so dominant it seemed like he may be right. But there were some weeds growing in the cracks: C, Smalltalk, Ada… an interesting time to be programming.Wwe settled on C, progressed to C++, progressed to Java. Then Java. Java. Java. Another monoculture! And then sometime around 2006 people got frustrated with Java and Rails was the crack in the dike that got mainstream programming talking about languages, and the interesting times are back. So the goal for today is to tour the languages Olsen has his eye on. Criteria Successful languages need something old, new, borrowed, blue. Old: the early days of Fortran involved inventing everything: nearly the first compiler, the runtime, documentation, invent the language, everything from scratch! It just doesn’t happen anymore, the standards are so high. So every language builds on another. Python built on Perl [he means ABC]. C borrowed from Fortran; C++ from C; Java from C/C++; Ruby borrowed from SmallTalk. New: A language needs to has to be “really better” to justify the time it takes to learn, resources to support. C wasn’t Fortran; C++ added objects to C; Java added objects to C (again) and was cross-platform; Ruby is dense and has some metaprogramming. Borrowed: Have to take good things from the surroundings. C lifted from everywhere and rolled them up; C++ found OO; Java “invented” the VM; Ruby stole liberally from SmallTalk. Blue: (stretches the analogy, but…) Thinking of IBM here, the support to get big. In other words, luck. Java had Sun; Ruby filled a desperate need. Curb appeal is also important, when you look at a house from the curb and instantly fall in love. SmallTalk didn’t have it. Python’s hamstrung by significant whitespace. Ruby is nice. People who come to a Saturday conference will be up for almost anything, but the 9-to-5 crowd has a vote here [die scum]. The Contenders Groovy Basically Java (100% compat) without the noise. Probably the language Olsen knows best. Has Java, knows Ruby… but suffers from trying to combine static with age and complexity with dynamic fun stuff, “it’s not an easy marriage”. Classes are almost exactly Ruby. Judgement: There doesn’t seem to be quite enough new, it won’t displace Java. Scala JVM Based, not 100% Java compat. Functional and object-oriented. Static typing can be quite complex (and not just from the perspective of a Ruby programmer). Avoids state and side effects, which is nice for multithreading. Concurrency via Actors, which are concurrent processes with message queues that are the only way data flows. Harder to screw up the threading. Code samples look a fair bit like Java at first, but some of the typing can get imposing. Looks like it and Groovy are going to be in a deathmatch for “The Better Java”, mostly fighting about typing. Clojure JVM based but not like Java: it’s a Lisp variant! Real metaprogramming, dynamic typing, functional (well, mixed because Java stuff you call isn’t). Concurrency via STM, which is analogous to a database transaction. Samples… yeah, get your parenthesis on! And that’s the problem with Lisp, it doesn’t have curb appeal, the 9-to-5ers can’t get over it. Erlang Not JVM, has its own VM and consequently the most functional of the contenders. Dynamically typed, also uses Actors. Hot-swappable code, which is quite neat. Hello worldis simple. Factorial does pattern matching, which Olsen likes. The socket server example lacks curb appeal. Audience pipes up to gripe that Erlang has C-like “array of bytes” for strings. Reia Runs on Erlang VM and looks like Ruby. Quite a lot, really… well, except that nifty pattern matching. Not purely functional. Smells like how Ruby was great because it took great things from the unsexy SmallTalk, maybe Reia could do it for Erlang. Conclusion So, what will things look like in 30 years? Olsen have a friend who, since 1980, has only coded in Fortran. Living in interesting times is a choice. What is yours? Q&A Q: No JavaScript? A: “I vacillated.” Ultimately decided the audience probably knew it better. Considers it a serious and interesting language because of Rhino. Q: Aside from obvious curb appeal problem, Haskell? A: Fails a test I didn’t get into: when the people who know it spend their time arguing what a given program means, stay away. Q: .Net? A: “Uhhhh…’ (Audience: …that’s still the west side of hell!) Olsen said he’s just not familiar with the platform.