[ Try Mouth Synth | View source code on GitHub ]

In my previous experiments, I integrated hand pose detection in my sketch. Hand-driven interaction feels magical, but also somewhat conventional. For this new series, I wanted to try something less conventional—mouth-driven interaction.

For this series, I wanted to create a simple interface for “drawing” melodies on the screen. I started with a mouse interaction where clicking and dragging the mouse drew a sequence of linked nodes along the path of the cursor movement. It worked much like how you would expect any drawing app to work. The melody would start playing as soon as you release your mouse. The nodes were sequentially highlighted as their corresponding notes were played.