Developer Day Notes 6: Browser-Based Visual Programming with Lily and the Monome

by Justin Marney

Goals: See amusic generator built using a Visutal Programming Language. Learn when, how, and why to use a VPL. Use the VPL Lily to mash the DOM, JS, and the Monome together. Party.

The Monome is a physical computing interface with open-source hardware, hand-made. [damn cool]

First, a demo based on Conway’s Game of Life. (Other VPLs: Max MSP, Pure Data). Idea! Use the Game of Life to generate MIDI. Create the start state, evolve every n ms, interpret state as a series of MIDI notes, display on the Monome, and reset to the start state every x iterations. Then, make it multi-track.

(Awesome demo of light and msuic commences.) It involves setting patterns on the Monome and tweaking volume/samples on the monitor in Open Sound Control, a VPL. Starts simple and grows quickly complex as the patterns evolve. And the msuic is quite listenable techo/glitch.

Thisis a “fringe technology”. It allows you to quickly create a working application with a usable interface, but you probably won’t be able to apply any of it to your day job.

Toolbox: Monome (optional), VPL Lily (like PureData but in the browser), and “A community of smart people and open-source software”.

The Monome is totally programmable. It’s a box of buttons, lighting up is decoupled: pusing a button sends a signal, your code could light it up instantly or not as your app decides. Apps exist for sound/midi generation, pixel animators, games, application control surfaces, video controls.

“It’s minimal, but not so minimal you can’t get something done with it… these constraints fuel your creativity.”

VPL programs are like big state machines. Boxes implement functionality and arrows connect boxes to pass data. No syntax to learn, often highly intuitive, and programs can be manipulated while they are running. Mostly they’re used by people doing live art (music/video/performance).

(A demo of Lily.) It installs as a Firefox Add-On. Shows a small bit of code, a “patch”, that creates an array of names and maps them through string interpolation and print to print “Hello, X”.

Because it’s in your browser, you can make the browser open a URL, analyze the DOM, even alter the DOM.

Demo: open Craigslist, shuffle all the <a> tags and write them back to the DOM. “So this is crazy. Why would you ever do anything like this? Well, you wouldn’t.” This demo is not especially useful, but you could pull in Javascript libraries or scripts, maybe set it up so that choosing a word opens a lightbox filled with related pictures from Flickr.

You can even save patches to create a standalone Firefox Add-On.

(Shows js code to create Lily boxes.) Looks very clean and straightforward. You can use HTML do display things (in this case a checkbox form input) in your Lily patch.

Open Sound Control is a network communication protocol that uses URL-style naming. “Who’s heard of OSC? One guy? Yeah, that guy uses his computer to make music.” MIDI is nice, but showing its age and can be slow. OSC is a replacement.

Monome OSC protocol sends simple “/name/led x y 1” to turn on or “…0” to turn off a light, similarly simple for keypresses.

“Weird mashup” demo: turn the DOM into a series of sounds so you could “play” a web page. Example page: startrek.com.

Resources:

Q&A:

Q: There’s monome support for Impromptu. A: There’s monome support for anything open.

Q: I’ve had problems with copy and paste in code. How do VPLs deal with that? A: You can create js classes and reuse those, and create a box that contains a patch to pull it in many times. Overall, it feels a lot like functional programming.