This is a living document and is subject to change.
- IMNY-UT 224 Introduction to Machine Learning for the Arts
- Interactive Media Arts (IMA), Tisch School of the Arts, New York University
- 370 Jay Street, Room 412, Brooklyn, NY 11201
- Tuesday/Thursday 6:00 PM – 9:30 PM (7/9/2024 – 8/15/2024)
- Jack B. Du
- jackbdu@nyu.edu
- Office Hours: Monday/Wednesday 2:00 PM – 4:00 PM
- Introduction to Machine Learning
- p5.js Review Session
Thursday 7/11
- Image Classification
- ml5.js Coding Session
Tuesday 7/16
- Transfer Learning
- ml5.js Coding Session
Thursday 7/18
- Pre-trained Models
- ml5.js Coding Session
Tuesday 7/23
- Training a Neural Network
- ml5.js Coding Session
Thursday 7/25
- Convolutional Neural Network
- ml5.js Coding Session
Tuesday 7/30
- Text Generation
Thursday 8/1
- Image Generation
Tuesday 8/6
- Final Project Proposals
Thursday 8/8
- Final Project Prototypes
Tuesday 8/13
- Final Project Testing
Thursday 8/15
- Final Project Presentations
Please read and review the ITP/IMA Community Commitments. The Community Commitments will be reviewed and discussed as part of the course introduction.
The ITP/IMA Community Commitments is an evolving work-in-progress document that establishes and communicates the commitment of the ITP/IMA community to uphold a key set of standards and obligations that aim to make ITP/IMA an inclusive and welcoming environment.
An introductory course designed to provide students with hands-on experience developing creative coding projects with machine learning. The history, theory, and application of machine learning algorithms and related datasets are explored in a laboratory context of experimentation and discussion. Examples and exercises will be demonstrated in JavaScript using the p5.js, ml5.js, and TensorFlow.js libraries. In addition, students will learn to work with open-source generative models including large language models and image generation models. Principles of data collection and ethics are introduced. Weekly assignments, team and independent projects, and project reports are required.
At the completion of this course, the student will:
- Develop an intuition for and high level understanding of core machine learning concepts and algorithms, including supervised learning, unsupervised learning, reinforcement learning, transfer learning, classification, and regression.
- Be able to apply machine learning algorithms to real-time interaction in media art projects using pre-trained models and “transfer learning” in JavaScript and related tools.
- Learn how to collect a custom dataset to train a machine learning model and
- Develop a vocabulary for critical discussions around the social impact and ethics of data collection and application of machine learning algorithms.
- Become familiar with the current landscape of new media art generated from machine learning algorithms. Understand how to use a machine learning model to generate media: words, sound, and images.
You will need a modern laptop (4 years old or younger is a good rule of thumb). Most required software is freely available. The department has all required commercial software installed on laptops available for checkout.
There is no textbook for the class. Readings and videos will be assigned on the individual session notes pages.
Grades for the course will follow the standard A through F letter grading system and will be determined by the following breakdown:
- 25% Participation
- 50% Assignments (including reading responses and other written work)
- 25% Final project
At most two (2) unexcused absences will be tolerated without effect to your grade. Any more than two (2) unexcused absences will result a lowering of your final grade by one whole grade for each unexcused absence. For example, three (3) unexcused absences will result in your highest possible grade being a B instead of an A. Four (4) unexcused absences will result in your highest possible grade being a C and so on. Six (6) unexcused absences would result in an automatic F for the course. Two (2) late arrivals will count for one (1) absence.
This class will be highly participatory. You are expected to contribute to discussions, engage in group work, give feedback to your peers, and otherwise fully participate in class.
Recreational use of phones, music players, video game systems and other devices are forbidden. Laptops and tablets are allowed for note taking, in class work, as well as relevant research only. Activities not related to the class, such as recreational use of the internet, including all social media websites, email and instant messaging, game playing, and work for other classes, will not be permitted. Such activities are disrespectful to the instructor and distracting to others. Your laptop should always be closed whenever a fellow student is presenting.
Classes will be a mixture of lecture, discussion, hands-on tutorials, homework review, presentations, and group work. You will need to come to class prepared with a laptop and any other supplies specified for that class.
The course will be once (1) per week for three and half (3.5) hours for a total of six (6) weeks.
There will be regular assignments that are relevant the class material. These assignments must be documented (a written description, photos, screenshots, screen recordings, code, and video all qualify based on the assignment) on a web platform such as a blog or google doc. You are required to link to your assignment from the course repo (you may choose to use a privately shared google doc or password protected website if you prefer.) The due dates are specified on the assignment page.
It is expected that you will spend 15 to 20 hours a week on the class outside of class itself. This will include reviewing material, reading, watching video, completing assignments and so on. Please budget your time accordingly.
Each assignment will be marked as complete (full credit), partially complete (half credit), or incomplete (no credit). To be complete an assignment should meet the criteria specified in the syllabus including documentation. If significant portions are simply not attempted or the assignment is turned in late (up to 1 week) then it may be marked partially complete. If it is more than a week late, not turned in, or an attempt isn’t made to meet the criteria specified it will be marked incomplete.
Responses to reading and other written assignments are also due in class one week after they are assigned and must also be submitted via the class website. Written assignments are expected to be 200 to 500 words in length unless otherwise specified. Grading will follow the same guidelines as above; on time and meeting the criteria specified will be marked as complete. Late (up to 1 week) or partially completed work will be given half credit. Work that is more than a week late, not turned in, or fails to meet the criteria specified will be given no credit.
Plagiarism is presenting someone else’s work as though it were your own. More specifically, plagiarism is to present as your own: A sequence of words quoted without quotation marks from another writer or a paraphrased passage from another writer’s work or facts, ideas or images composed by someone else. More information can be found on Tisch’s page regarding Academic Integrity (https://tisch.nyu.edu/faculty/academic-integ).
(The following is adapted from Golan Levin’s Interactivity and Computation Course (Fall 2018) at Carnegie Mellon University.)
You must cite the source of any code you use. Please note the following additional expectations and guidelines:
-
Check the License. When using others' code, pay attention to the license under which it has been released, and be certain to fulfill the terms and requirements of those licenses. Descriptions of common licenses, and their requirements, can be found at choosealicense.com. Some licenses may require permission. If you are confused or aren’t sure how to credit code, ask one of the course instructors and make your best good faith effort. Not properly citing code sources is grounds for a 0 on an assignment.
-
Use Libraries. The use of general, repurposable libraries is strongly encouraged. The people who developed and contributed these components to the community worked hard, often for no pay; acknowledge them by citing their name and linking to their repository.
-
Be Careful. It sometimes happens that an artist places the entire source code for their sketch or artwork online, as a resource from which others can learn. Assignments professors give in new-media arts courses are often similar (e.g. "Clock"); you may also discover the work of a student in some other class or school, who has posted code for a project which responds to a similar assignment. You should probably avoid this code. At the very least, you should be careful about approaching such code for possible re-use. If it is necessary to do so, it is best to extract components that solve a specific technical problem, rather than those parts which operate to create a poetic experience. Your challenge, if and/or when you work with others' code, is to make it your own. It should be clear that downloading an artwork from someone's GitHub and simply changing the colors would be disgracefully lazy. And doing so without proper citation would be outright plagiarism.
Academic accommodations are available for students with documented disabilities. Please contact the Moses Center for Students with Disabilities at 212 998-4980 for further information.
Your health and safety are a priority at NYU. If you experience any health or mental health issues during this course, we encourage you to utilize the support services of the 24/7 NYUWellness Exchange 212-443-9999. Also, all students who may require an academic accommodation due to a qualified disability, physical or mental, please register with the Moses Center 212-998-4980. Please let your instructor know if you need help connecting to these resources.
Tisch School of the Arts is dedicated to providing its students with a learning environment that is rigorous, respectful, supportive and nurturing so that they can engage in the free exchange of ideas and commit themselves fully to the study of their discipline. To that end Tisch is committed to enforcing University policies prohibiting all forms of sexual misconduct as well as discrimination on the basis of sex and gender. Detailed information regarding these policies and the resources that are available to students through the Title IX office can be found by using the following link. https://www.nyu.edu/about/policies-guidelines-compliance/equal-opportunity/title9.html
This syllabus is adapted from the work of previous instructors (Daniel Shiffman, Yining Shi) who have generously shared their materials.