- March 31, 2017
Visually-Induced Auditory Synesthesia
This project is intended to be self-directed with a poetic approach/interpretation to the term “trace” as a trigger word. “Trace”, as a group of visual images with regular/irregular intervals just as footsteps, can be translated into unique, distinct auditory modality as in sounds, beats, and rhythms by the multi-modal nature of it (e.g., a loop of sound, repetitive visual images, tactile impressions, etc). In this context, users can visualize loops of sound with visual patterns that they create.
Demonstrating Interaction : Due to complexity of the interactive synesthesia experience, I decided to use physical computing for demonstrations. Ideally, it also takes linear mechanical movement as the prototype has to read loops of visual images, image by image, in order to audioize them.
Breaking down the system into testable pieces
Testing out several electronic prototypes, I have come down to a simple hardware that allows you to draw and audioize/auralize your trace as if you create your very own joyful beats! Tests have been carried out, using the additive manufacturing/prototyping method so it gets optimized for quick revisions and re-work for frequent updates in code.
Data Translation
Made out of acrylic sheets, the main prototype has a tracking sensor installed on the bottom of its body, so it recognizes/reads data from the visual patterns you create, translating them into loops of sound once physically moved in a linear way. The prototype contains a tracking sensor as an input, buzzer as an output, Arduino Nano, and a bread board to operate the playful interaction (The tracking sensor has a capability of tracing and tracking images/objects with infrared LEDs).
- Vimeo Link 01 Testing (Pattern A)
- Vimeo Link 02 Demo (Pattern B)
- Vimeo Link 03 Interaction (Complete)
Technical Drawing : The prototype (a works like prototype) is designed to fit on A3 and A4 size papers as the rails on both sides are 400 millimeters in width, covering visual data within the range.
