Intro-ML-Arts-IMA-Summer24

Text Generation

[ Slides | Sketches ]

Objectives

• Learn about sequential data and Markov Chains.
• Learn about Recurrent Neural Network (RNN) architectures.
• Learn about transformer architecture.
• Learn to work with large language models (LLMs) in JavaScript.

Lecture Notes

Tools

• Tokenizer
  • OpenAI Tokenizer Demo
• Ollama
  • Ollama Open-Source Large Language Model Library
  • Ollama JavaScript Library
  • Unofficial Ollama JS Library
• OpenAI API
  • OpenAI API Reference
  • OpenAI API Playground

Related Projects

Text

• ITP Course Generator by Allison Parrish.
• Sunspring by Oscar Sharp, Ross Goodwin, et al.
• Writing with the Machine by Robin Sloan.
• Let’s Read a Story by Itay Niv.
• 10 things artificial intelligence did in 2018 by Janelle Shane.

Drawing

• Double Agent by Simon Biggs.
• Four Experiments in Handwriting with a Neural Network by Shan Carter, et al.
• Handwriting Generation with RNN and p5.js by @hardmaru.

Music

• Magenta: Make Music and Art Using Machine Learning by Google AI.
• RNN for generating Baroque Music video by @carykh.

Coding

• Using ChatGPT to Implement Sol LeWitt’s Wall Drawings by Amy Goodchild.

Code Example

Note: Ollama examples below can only be run locally in conjunction with Ollama.

Markov Chain

• Markov Chain Demo

Ollama

• Ollama - Minimal Example
• Ollama - Click for New Response without Context
• Ollama - Predetermined Prompts with Context
• Ollama - Chat Completion with Context

OpenAI

• OpenAI API - Chat Completion without Context
• OpenAI API - Chat Completion with Context
• OpenAI API - Image Generation

Supplemental Materials

Markov Chains

• Markov Chains by Victor Powell and Lewis Lehe.
• N-Grams and Markov Chains by Allison Parrish.

RNNs and LSTMs

• Understanding LSTM Networks by Christopher Olah.
• The Unreasonable Effectiveness of RNNs by Andrei Karpathy.
• Recurrent Neural Networks & LSTMs by Lenny Khazan and Rohan Kapur.

Transformers

• But what is a GPT? Visual intro to transformers by 3Blue1Brown.
• Attention in transformers, visually explained by 3Blue1Brown.
• The Foundation Model Transparency Index from Center for Research on Foundation Models at Stanford University.

Video Tutorials

Note: ml5.js tutorials below were taught using an older version of ml5.js, refer to the ml5.js Resources Wiki page for more information.

Text

• Markov Chains - Part 1 - video tutorial by Daniel Shiffman.
• Markov Chains - Part 2 - video tutorial by Daniel Shiffman.
• Context-Free Grammar - video tutorial by Daniel Shiffman.
• What is word2vec? - Programming with Text - video tutorial by Daniel Shiffman.

Drawing

• Sketch-RNN Snowflakes with ml5.js - video tutorial by Daniel Shiffman.
• Interactive Drawing with Machine Learning Model (SketchRNN) - video tutorial by Daniel Shiffman.
• RDP Line Simplification Algorithm - video tutorial by Daniel Shiffman.

Assignment 7

1. Read What Can Machine Learning Teach Us About Ourselves?, interview with Emily Martinez, ml5.js Fellow 2020.
2. Read The Subtext of a Black Corpus, in conversation with ITP research fellows Nikita Huggins & Ayodamola Okunseinde by Ashley Lewis.
3. Emily Martinez proposes a set of questions to ask related to working with a corpus of text data. Pick one (or two) of the questions to reflect on as you respond to the above two readings:
   • How can we be more intentional about what we build given the current limitations, problems, and constraints of ML algorithms?
   • How do we prepare datasets and set up guidelines that protect the bodies of knowledge of our communities, that honors lineage, that upholds ethical frameworks rooted in shared, agreed-upon values?
   • How do we work in consensual and respectful ways with texts by marginalized authors that are not as well-represented, and by virtue of that fact alone, much more likely to be misrepresented, misappropriated, or misunderstood if we are not careful?
   • How well can we ensure that the essence of these texts doesn’t dissolve into a word-soup that gets misconstrued?
   • Given that so many of the existing “big data” language models are trained with Western texts and proprietary datasets, what does it even mean to try to decolonize AI?
   • Who do we entrust to do this work?
   • How do we deal with credit and attribution of our new creations?
   • How do we really do ethics with machine learning?
   • How do we get through this whole list of concerns and still build AI that is fun, respectful, tender, pleasurable, kind?
4. Document your response to the readings in a blog post and add a link to the post on the Assignment 7 Wiki page.
5. Review the final project proposal guidelines and post your final project proposal and slides on the Final Proposal Wiki page.

This site is open source. Improve this page.