For the past two years, I have been doing lots of computational sketches that focused on visuals. This year, I am enrolled in a graduate course at ITP called the Code of Music, taught by Luisa Pereira. I finally ran out of excuses for not experimenting with sound.

For this first experiment, we were prompted to build a simple audio visual instrument in p5.js.

Coincidentally, I have just recently started learning shader in p5.js following this very well-structured course by Elie Zananiri. Before even writing any shaders, I was already amazed how I could use vertices to define a 3d mesh constructed by a triangle strip, specify a unique color at each vertex, and the 3d mesh will be filled with color gradients that interpolate these colors. With this idea in mind, I started experimenting with a donut-esque mesh by altering its level of details, animating its vertex colors, and applying oscillation to its vertices.

Experiment on January 28, 2024
[ View post on Instagram ]

This served as a good foundation for my visual. Now it is time to choose audio.

I did not want my instrument to sound like a conventional instrument, so I was planning to record some random sounds. It came to my realization that I did not have time to identify sounds that I wanted to record, record them, and then edit them. So I turned to freesound.org. I was searching for water sounds and came across this very good quality paddle stroke sound pack.

This sound pack includes a decent collection of short paddle stroke sounds that I could use.

Following this sample sketch from Luisa, I did some quick sketches to familiarize myself with tone.js.

Once acquinted with tone.js, I finally linked my audio to my visual.

Experiment on January 31, 2024
[ View post on Instagram | View source code in p5.js Web Editor ]

In this sketch, I am wrapping the audio waveform around the mesh and applying it as a factor to the radius. In addition, the amplitude of the audio is applied as a factor to the amplitude of the oscillation.

The first version incorporates a keyboard interaction. Later, I thought it would be more interesting to use a mouse to interact as if the mouse is paddling the water.

In the live demo below, you can click and drag your mouse around, and the water sound reacts to how fast you move your mouse.

Experiment on February 2, 2024
[ View post on Instagram | View source code in p5.js Web Editor ]