Color Code

The Color Code project is a set of experiments, with Jay Silver, in using the colors of everyday objects to control things on a computer. The four videos below show the evolution of our prototypes, from the "Twinkle" system that turns colors into music, to the increasing complexity of "Color Code," "Scratch Color Code," and finally "Programming with Color."

The vision begins with "Twinkle": a sensor that turns a sequence of colors into a melody, so that a drawing, a lego structure, or stripes on a shirt can become a musical score.

Our next prototype, "Color Code," generalizes this idea so colors of objects can control action on a screen, from a pong game to a music sequencer to an artificial life simulation.

"Scratch Color Code" integrates Color Code into the Scratch programming environment, so that people can use the friendly graphical programming environment to create their own systems that interact with the colors of objects. The examples we made include a cookie monster game, a platform game with platforms made from actual leaves, and a candy drum machine.


Finally, we demonstrate our furthest extension of the concept: "Programming with Color." Here, we use our Scratch Color Code to create a tiny programming language, in which colors represent commands. The little character moves across the screen, drawing a line. At the bottom of the screen, a black arrow slides right to left, reading a "program" made up of colored objects. When the arrow touches something red, the character turns right. When the arrow touches green, the line that the character is drawing changes color. Yellow means turn left, and blue means start over. This is a tangible programming language, because the commands are represented by objects, which can be picked up and rearranged, even while the program is running. Unlike most tangible programming languages, the commands can be represented by any object with the right color. And unlike other programming languages, the commands have a continuous quality: the wider the red object is, the longer the character spends turning to the right. These properties give the language some interesting affordances for playful exploration.