[ Try Bug Musicans ]

This series of experiments started with something completely unrelated to music. I remember seeing ragdoll characters commonly used in video games and artworks either to invoke a surreal feeling or for a comedic effect. I wanted to try implementing it in p5.js with the help of a physics library.

[ Try Mouth Synth | View source code on GitHub ]

In my previous experiments, I integrated hand pose detection in my sketch. Hand-driven interaction feels magical, but also somewhat conventional. For this new series, I wanted to try something less conventional—mouth-driven interaction.

For this series, I wanted to create a simple interface for “drawing” melodies on the screen. I started with a mouse interaction where clicking and dragging the mouse drew a sequence of linked nodes along the path of the cursor movement. It worked much like how you would expect any drawing app to work. The melody would start playing as soon as you release your mouse. The nodes were sequentially highlighted as their corresponding notes were played.

In this series of experiments, I started with very basic algorithmically-generated rhythms coupled with simple visual elements.

While we were exchanging ideas of what we could do with Tone.js and p5.js, Henry Chen came up with this idea of a node-based system for interacting with music in a virtual 3D environment, in which moving one node could simultaneously affect multiple other nodes that affect the music. We agreed that creating a 2D version first could serve as a very good starting point.

For the past two years, I have been doing lots of computational sketches that focused on visuals. This year, I am enrolled in a graduate course at ITP called the Code of Music, taught by Luisa Pereira. I finally ran out of excuses for not experimenting with sound.