Intro-ML-Arts-IMA-Summer24

Body Models

[ Slides | Sketches ]

Objectives

• Learn about TensorFlow Pose Estimation model and how it works.
• Learn about TensorFlow Body Segmentation model and how it works.
• Learn the differences between MoveNet and BlazePose and how to choose between them.
• Learn the differences between BodyPix and SelfieSegmentation and how to choose between them.
• Learn to work with ml5.bodyPose and ml5.bodySegmentation.

Lecture Notes

Tools

• ml5.bodyPose Reference
  • ml5.bodyPose Source Code
  • TensorFlow.js Pose Detection Source Code
  • MoveNet.SinglePose Model Card
  • MediaPipe BlazePose CHUM 3D Model Card
• ml5.bodySegmentation Reference
  • ml5.bodySegmentation Source Code
  • TensorFlow.js Body Segmentation Source Code

Related Projects

• PomPom Mirror by Daniel Rozin.
• The Treachery of Sanctuary by Chris Milk.
• Moving morrir from Google Creative Lab.
• Touch Type by Richard Yee.
• PoseNet Sketchbook by Maya Man.
• AR Body Filters by Daria Sazanovich.
• Dance with Bubbles by Sihan Zhang.
• nixel-Body from CMU Interactivity & Computation.
• Web Olympic by Vibert Thio.

Choreography

• Body, Movement, Language: A.I. Sketches with Bill T. Jones from Google Creative Lab.
• ballet rotoscope by Euphrates.
• Messa di Voce by Golan Levin, Zach Lieberman, Jaap Blonk, and Joan La Barbara.

Music

• Body Synth by Use All Five & Google Creative Lab.
• Semi-Conductor from Google Creative Lab.
• This Is Not A Theremin by Guillermo Montecinos and Sofía Suazo.
• Sidewalk Orchestra by Cristóbal Valenzuela.
• Pose Music by Tero Parviainen.

Animation

• Pose Animator by Shan Huang.
• Puppet Parade - Interactive Kinect Puppets by Design I/O.

Code Examples

Core Functionalities

• BodyPose - MoveNet Keypoints
• BodyPose - MoveNet Skeleton
• BodyPose - BlazePose Keypoints
• BodyPose - BlazePose Skeleton
• BodySegmentation - Mask Background with SelfieSegmentation
• BodySegmentation - Mask Person with SelfieSegmentation
• BodySegmentation - Mask Body Parts with BodyPix
• BodySegmentation - Mask Select Body Parts with BodyPix

Creative Applications

• BodyPose - Drawing with Nose
• BodyPose - Trailing Effect
• BodyPose - Cat or Dog
• Additional examples from Yining Shi (Note: these demos were created with an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information.)

Supplemental Materials

• Feminist Data Set, a tool kit to interrogate every step of the AI process by Caroline Sinders.
• The Future Is Here!, a commission for The Photographers’ Gallery which examines the process of dataset creation by Mimi Ọnụọha.
• Ted Talk: The era of blind faith in big data must end by Cathy O’Neil.
  • Cathy O’Neil’s Risk Consulting & Algorithmic Auditing firm.

Technical Blog Posts

• Real-Time Human Pose Estimation in the Browser with TensorFlow.js by Dan Oved, with editing and illustrations by Irene Alvarado and Alexis Gallo. [ Original Paper ]
• Next-Generation Pose Detection with MoveNet and TensorFlow.js by Ronny Votel and Na Li.
• 3D Pose Detection with MediaPipe BlazePose GHUM and TensorFlow.js by Ivan Grishchenko, Valentin Bazarevsky, Eduard Gabriel Bazavan, Na Li, Jason Mayes.
• Introducing BodyPix: Real-time Person Segmentation in the Browser with TensorFlow.js by Dan Oved and Tyler Zhu, with editing by Irene Alvarado.
• Body Segmentation with MediaPipe and TensorFlow.js by Ivan Grishchenko, Valentin Bazarevsky, Ahmed Sabie, and Jason Mayes.
• Pose Animator - An open source tool to bring SVG characters to life in the browser via motion capture by Shan Huang.

Video Tutorials

Note: ml5.js tutorials below were taught using an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information.

• ml5.js Pose Estimation with PoseNet - video tutorial by Daniel Shiffman
• ml5.js: Pose Classification with PoseNet and ml5.neuralNetwork() - video tutorial by Daniel Shiffman
• ml5.js: Pose Regression with PoseNet and ml5.neuralNetwork() - video tutorial by Daniel Shiffman

Assignment 3

1. Read Mixing movement and machine by Maya Man.
2. Read Humans of AI by Philipp Schmitt.
3. Pick one of the models above (BodyPose or BodySegmentation), following the examples above and ml5.js documentation, experiment with controlling elements of a p5.js sketch (color, geometry, sound, text, etc) with the output of the model.
4. Read Open Sourcing the Origin Stories: The ml5.js Model and Data Provenance Project by Ellen Nickles and reflect on the the following questions:
   • What questions do you still have about the model and the associated data? Are there elements you would propose including in the biography?
   • How does understanding the provenance of the model and its data inform your creative process?
5. Document your response to the readings as well as your p5.js sketch in a blog post and add a link to the post and your p5.js sketch on the Assignment 3 Wiki page. In your blog post, include visual documentation such as a recorded screen capture / video / GIFs of your sketch.

This site is open source. Improve this page.