Weekend webcam project: playing with the reacTIVision framework

Last weekend, I stranded on the reacTIVision site by coincidence. It took me some time to figure out what the reacTIVision open source project is all about, but once I did my weekend project was settled! 🙂

What does reacTIVision do?

This is what their site says: reacTIVision is an open source, cross-platform computer vision framework for the fast and robust tracking of fiducial markers attached onto physical objects, as well as for multi-touch finger tracking. My first reaction was “what the … is a fiducial marker?”. A fiducial marker turns out to be some sort of graphical symbol that is associated with a unique ID. The reacTIVision code uses the “amoebe fiducials” by default, which are cute amoebe-like figures. There are 180 of these amoebe fiducials, and each one of them is associated with a unique id (0 through 179).

a fiducial

So what does the reacTIVision framework do with these symbols? The software will analyze the image captured from your webcam, and locate any amoebe fiducials in it. The software is able to give the location of the symbol as wel as its orientation (rotation).

The software will write all this information (what symbols found, their location and their orientation) on a UDP port. This UDP port is not a real port on your computer but just a number that identifies where the information is written. Another program (custom written by you) can then read in this information by connecting to that port.

 Why is it cool?

On the software page of their site, the reacTIVision team provides not only the reacTIVision software itself, but also a demo application that uses this information. This demo application is available in a lot of different programming languages (C++, Java, C#, Processing, Pure Data, Max/MSP, Quartz Composer and others) so you can just use your prefered language to build your own cool application!

A commercial application (not open source) is the reactable (click here for a basic demo video), a cross-breeding between a touch table and a synthesizer, which looks like a really really really cool way to make music. Some of the symbols generate a tone (e.g. a sine wave) and other blocks will influence this tone (e.g. its frequency) as soon as they are close enough to the tone-block.

You don’t have to build a touch table for your own projects (altough it gives the coolest results I guess), you can just hold the symbols in front of a webcam too.

My own application

I built a simple test application that associates the fiducials with a loop sound. When I show a fiducial to my camera my program starts playing the loop so you can compose a simple track this way. I wanted to program it in Java at first, but the developing of the graphical output went way too slow to my liking, so I switched to Processing. If you don’t know this language (I didn’t) already, it really is worth checking out. It is targetted at artists so it is really easy to create fancy graphical output. It is built upon Java so it is even possible to combine it with Java code!

My own reacTIVision application 🙂

My application is shown in the above image. When I show a fiducial to the camera, it is detected and plotted in the application. A sound loop starts playing (I use the Minim library that is included in Processing for this) and the sound wave is plotted next to the symbol. The code can be downloaded from here. The sound loops were downloaded from http://www.freesound.org/packsView.php. I also experimented a bit with the filter possibilities of the Minim library. If you uncomment a line in my code, a bandpass filter depending on the fiducial’s position will be applied to the loop!

The code is quick draft code and certainly not error proof, so use it as your first steps into TUIO and Processing but be aware of all the things that could be coded better :p

My test setup

You can show your fiducials to your webcam and all will work well as long as you hold the fiducial pointing straight at the webcam. You can construct a simple setup by hanging a camera with a little tripod between some books like in the image below:

A minimal camera setup

 A better solution would be to construct a transparant table. I constructed a simple setup with two wooden IKEA boxes and some plexi-glass as shown below:

A more elaborate (but still very far from professional) setup :p

You can see the camera peeping through the hole below. The lamp and the white paper on the floor of the box are an attempt to improve the performance. If you can afford a projector you can even create a real multitouch table like in the reactable project!

Existing applications

You can find a lot of reacTIVision projects on the web. Most of them use it to interact with a sound producing application like e.g. Abbleton Live. Some other creative uses of the reacTIVision framework:

• The garage geeks tangible laptop project tries to build an alternative, portable and open source version for the reactable by using the laptop screen with a webcam installed above it instead of a touch table. Their code is available on google code, and a discussion board is available on google groups. Their code is written in C++.
• The “table explorer” is a simple project that uses the reacTIVision to display information to the user. http://www.hogtownconsulting.com/archives/tag/reactivision/
• The grafiti project (http://grafitiproject.wordpress.com/) is a gesture recognizer built on top of the reacTIVision by Alessandro De Nardi.
• Some cool projects at professor Kirsty Beilharz’s page also use the reacTIVision engine.
• This page also contains some experiments with reacTIVision.
• ReacTable Role Gaming Map Editor creates game levels with the reacTIVision
• This video illustrates using reacTIVision as an input to a microcontroller. This could be extended to e.g. switch your lights on and off by waving a symbol to your computer 🙂
• This is a cool game
• Live Composer associates notes with the fiducials to compose a melody
• Very cool layout on this tape-player project!
• A color mixer
• Tracking a robot’s location and feeding it back to the robot. This could be very useful while training e.g. a neural network controller for the robot.

So that’s it for this post.. If you know of some other cool reacTIVision projects, or (even better) have created your own cool application, let me know in the comments!