Quintilian in Action from Ritesh L on Vimeo.

Quintilian is a framework for hand based gestural interaction in immersive environments. It was implemented to take advantage of the existing real time motion capture system in the AlloSphere, focusing on the psycholinguistic and ergonomic aspects of everyday human interactions. The framework is intended to provide a basis for building a human centric interaction space derived from meaningful communication of gestural discourse (semiotic values) and manipulative properties of physical interaction (ergodic values). These principles are applied to achieve interaction tasks in immersive environments for navigation within virtual worlds and manipulation with their objects. A Device Server plugin was also implemented for the framework to make it easier for existing applications to obtain a much more natural mode of interaction than currently exists.

Quintilian was my Master’s project and the accompanying thesis can be found here: Quintilian by Ritesh Lala.

Virtual Synth is an experimental synthesizer for your iPhone/iPod touch. You can generate sine tones and square waves from a wide range of frequencies. Experiment with different beat frequencies to generate modulating signals. A minimal control interface and engaging graphics make it even more fun to use.

Interface:
Touch to create an instrument anywhere on the grid. The labels display frequency values. You can select a different instrument/frequency and tap on an existing instrument to change it. The beat frequency for modulating waves increases diagonally. Tap with two fingers at any time to remove all instruments.

Controls:
X: to toggle frequency labels.
+/-: to toggle adding or removing instruments.
F: to toggle slider display to change frequency range.
sin: to add a sine tone.
mod: to add a modulating signal.
sqr: to add a square wave.

Virtual Synth was created as a class project at the Media, Arts and Technology program of UCSB in the Spring of 2011. It was realized with constant help and encouragement from Charlie Roberts and Lance Putnam’s generic synthesis library- Gamma.

Interactive Installation @ MAT End of the Year Show 2011. (June 9, 2011)

Animus is an interactive multimedia installation inspired by Quorum Sensing in bacteria. It stands as a metaphor for the governing spirit of the system, a collective brain if you will, for cell to cell communication in microbes which in turn defines its gene expression and the collective behaviors that emerge from it. Micro organisms use this kind of communication constantly to check for their population density and crossing a certain threshold display behaviors that vary from bioluminescence and toxic secretion to sporulation and conjugation.

Animus places the user as a controller for this system, where how they choose to interact with it defines its outcomes. The user gets a continuous feedback of the threshold required to display a certain behavior (bioluminescence in this case), affecting how they interact with it, making them more a part of the system then its controller eventually.

Installation Description

The installation uses a Kinect to create a sensing field in front of the projection, making direct manipulation of the system possible unaffected by any background noise. The user needs to perform a certain gesture to identify them as the controller and start affecting the system’s behavior reflected by the audio interface and the display projection.

(Part of the MAT End of Year Show 2011, in Collaboration with Qian Liu)

Morphon is a visualization of Cellular Automata in 3D, for a specific rule set and the structures/patterns that emerge from it. Cellular Automaton is a discrete mathematical model studied in a number of advanced scientific fields. In its most elementary form it is a one dimensional row of cells, each with one of two possible states (ON or OFF) at any discrete moment of time, based on the states of its two neighboring cells and its previous self. There are rules governing this change of states over time, and thus emerges a complex mix of patterns sometimes periodic and sometimes highly stochastic from very basic building blocks.

This project investigates a subset of an extremely wide range of possibilities to visualize 3D Cellular Automata and explore the generative structures that emerge in the process. The user can interact with the emergent structures with an iPhone interface that communicates with the app using OSC messages. The sonification process is very basic in that it maps the number of living cells at any time to the number of instruments.

The TransLab is a space installed with IR trackers, multiple projectors and a 16-channel speaker system. This project involved  creating an immersive environment  where the user was surrounded by a fluid field that he could measure, interact with and manipulate with a sensor in his hand. The sensor was implemented with touchOSC on an iPhone. The fluid density values around the user were displayed on the iPhone as well as the projector screens and also played back as frequency modulations of sine tones, hence making it natural for the user to explore the space around him in trying to locate the fluid source. The sensor screen as seen by the user is displayed in the image below.

The 3D fluid field simulation was implemented by adapting Jos Stam’s 2D real-time fluid simulation algorithm for games  using 8000 points, with OpenGL in C++. Check out the video to see how the simulation looks in it’s mouse interactive version:

Real Time 3D Fluid Field Simulation from Ritesh L on Vimeo.

The possibilities to explore these structures were immense and so I skipped the next logical part where normally I would simulate 2D Cell Automata (a more specific example of which is “Conway’s Game of Life”) with 4294967296 (2^(2^5)) possibilities to explore. Instead I moved on to explore 3D CA. With millions of possibilities to explore again, it becomes crucial to constraint the system. So I explored Totalistic CA, where the rules are not updated based on state of a single cell, but based on the sum of their states, in a cell’s neighborhood. I ended up getting some interesting structures as shown in the images below. The sonification for this is rather primitive with the number of Cells being updated directly affecting the frequencies.

Since the system has many cells, and hence many states it was important to constraint it somehow when attempting to sonify it. Some attempts at creating generative music are following. The last video is an example of mapping Rule 110 on Pelog Scale to generate rythmic sounds, midway through which I dynamically increment/decrement the rules to change the structure.

Cellular Automata on Pelog Scale from Ritesh L on Vimeo.

( Continued on Visualizing Cellular Automata – III… )

The origins of Cell Automaton lie in Von Neumann’s simplification of the process of Kinematic Automata, a system designed to create self-replicating robots, due to Stanislaw Ulam’s insight on his methods. Though it became popular within a small computing community with John Conway’s “Game of Life”, it was Stephen Wolfram’s publication of “A New Kind of Science”, a book that explains how complex systems emerge from seemingly simplistic ones like Cell Automata, that reintroduced its concept as a thoroughly systematic investigation. The basics are very straightforward- you start with a set of initial states, iterate through all the cells, checking each cell’s neighborhood (a finite number of cells around it) and mapping its states to the rule being employed to calculate the next state of the cell. All the cells are updated once the rule is employed and then the process is repeated.

I started with Elementary Cellular Automata- 1D structure of cells, where each cell’s neighborhood is composed of itself, the cell on its right and the cell on its left, and there are only two possible states for each cell: ’0′ and ’1′. With this configuration you have a possibility of 256 (2^(2^3)) rules to govern the behavior. Interesting behaviors emerge when the evolution of 1D cellular automata is tracked for a number of iterations. The following images display some of the interesting rules. The major observation Wolfram made was how some structures were very orderly while some very stochastic in nature. Although some of the most interesting ones are with a combination of both, order and randomness in their structure, for example rule 110.

( Continued on Visualizing Cellular Automata – II… )

A. Face Recognition with Eigen Faces

B. Object Detection with Bag of Features

C. Homogeneous Transformation

D. Scale Invariant Feature Transform

E. Corner Detection (Harris Corner Detector)

Virtual Synth is an application of OpenCV for Processing. It creates a virtual frame on top of the camera stream that contains interactive objects. The square objects are updated on every frame of the stream to detect motion in that region, and as a result update the frequency of a Synth in supercollider to play different tones. Communication between Processing and Supercollider is done via oscP5 which uses Open Sound Control. GUI objects include sliders for changing contrast, brightness and threshold values for motion detection.

Check out the demo video:

Virtual Synth Behind the Scenes from Ritesh L on Vimeo.