/hcml_course

Primary LanguageHTML

Photo of man caught in cog{: .center-image }

By definition, Machine Learning provides software with a way to learn something that it was not explicitly programmed for. In practice, Machine Learning is often used to allow people to learn something they might not have been able to do otherwise (e.g. infer relationships from large scale high-dimensional data at potentially high speed) and allow computers to learn something that humans learn naturally (e.g.: speaking, hearing, seeing). These two broad application areas of Machine Learning not only have a large impact on human behavior, their success often depends on a nuanced understanding of human behavior and how people interact with technology (i.e.: sociotechnical behavior). Students of the Human-Centered Machine Learning course will help form a new way of understanding and practicing the application of machine learning through a series of readings, discussions, and a final project.

Introduction

Prerequisite Education

  • At least one Human-Computer Interaction or Design Thinking course
    • Specifically a familiarity of human factors and the emprical methods used to uncover them
  • At least one Machine Learning, Artificial Intelligence, or Data Science course
    • Specifically students should be comfortable applying methods for supervised and unsupervised learning in a software application of their design.
  • Ability to program at a level appropriate for EECS 349 (Machine Learning)
    • Specifically, you should be able to comfortably write code in Python without help from the TA or Professor

Learning Objectives

Students who complete this course satisfactorally will...

  • ...be comfortable discussing machine learning and artificial intelligence as part of a broader socio-technical system
  • ...have an understanding of how the behavior of artificial intelligence and machine learning algorithms as well as user task, intention, and understanding affect the overall user experience.
  • ...understand how to identify design constraints and optimization criteria from both the user experience and algorithm research in a domain.
    • In both cases, these may be more implicit than explicit requiring that students develop an eye for:
        1. translating UX research into design principles that can be addressed with tweaks to the learning algorithm.
        1. identifying assumptions that are made in the design of the learning algorithm and how they might (or might not) be challenged.

Assignments and Grading

Below are the requirements and expectations of the class with the respective grade proportions.

  • Final Project (40%)
  • Class Presentation and Leading Discussion (20%)
  • Reading Response (20%)
  • Attendance and Class Participation (20%)

HCML Project

Students will complete a final research paper that explores HCML in one of the following ways:

  • researching a system of the students' creation which integrates user-centered design principles into the design of the machine learning components of the system to address a user experience challenge or a novel user task
  • researching the design and implementation of an interactive machine learning system which might address some of the issues with automated learning systems discussed in class including (but not limited to) algorithmic bias, algorithmic accountability, and model transparency.
  • makes explicit through proof or demonstration an assumption used in the design of a learning algorithms which can result in one of the issues with AI and ML algorithms discussed in class (e.g.: algorithmic bias, concept evolution)

Students may also propose an alternative to one of these research paper formats.

The schedule and ruberic for the HCML project is as follows:

  • 5th week of class: Students submit an introduction and literature review and/or methods section by the
    • This will be reviewed by the instructor as well as two other students in the class as part of a paper feedback exchange
    • 5/5 points of the final grade for submitting a draft of reasonable enough quality to get helpful feedback
    • 2/5 points for submitting a draft which is not of reasonable enough quality to get helpful feedback
    • 0/5 points for not submitting
  • 7th week of class: Students submit a methods section along with working code
    • for an analytic project this generally means the jupyter notebooks or R Studio code describing the analytic process and working data pipeline.
    • For a systems project this will mean a prototype that is working well enough that the idea can be demonstrated even if further iterations are needed to effectively study that which will be the contribution of the final paper.
    • 5/5 points for a methods section which communicates the process well enough for the feedback to focus on the process and not the writing. For 5 points, the students will also have a data pipeline or system that is working well enough to begin providing good analytic or design feedback.
    • 4/5 points for a methods section which still requires feedback on how the method is communicated. For 4 points, the data pipeline or system should be demonstrable.
    • 3/5 points for submitting any methods section and code
    • Project
  • 9th week of class: Students submit a full draft of their project
    • 5/5 points for a draft which can yield good feedback
    • 3/5 points for including all complete sections
    • 1/5 points for submissions which do not include all major sections (introduction, methods, results, discussion, conclusion)
  • Final week: Complete submission

The HCML project will need to have a rough draft and a poster and/or demo completed by the 10th week of class for a poster/demo session that will be open to the public. The final draft will need to be completed at the beginning of finals week so that students can exchange and review each others' work.

Class Presentation and Leading Discussion

Students will lead discussion with a presentation twice in the quarter:

  1. One presentation will be leading a 45 minute discussion and presentation on research that is already included in the schedule. This presentation will also need to synthesize and incorporate reading responses from the other students.
  2. One presentation will be leading a 20 minute discussion and presentation on research, found by the student, that is related to the students' final project and enriches the conversation in human-centered machine learning. The purpose of this presentation is to facilitate an on-going conversation about how the practice and research of machine learning can be more "human-centered".

Reading Response

In order to better digest the readings, students will write a response to each weekly discussion prompt (provided in the reading schedule) by making appropriate citations and connections to the current week's readings and those of previous weeks. Students should feel free to cite other sources, but these should not be a replacement for the required readings of the week. This response will need to be uploaded by the Monday (subject to change) before class so that students will have a chance to read each others' responses and the presenter can integrate them into the weekly presentation. This exercise will provide students with the opportunity to get more comfortable with the writing and reflecting component of the class.

Class Attendance and Participation

In order to fully make the connections between the human-focused research and technology focused research in each week's readings it is important to participate in class discussion. Class discussion also helps students to practice communicating their understanding of the material to members of the class whose research focus and educational background skews toward human subjects and factors and those who skew toward computational research. Students will be required to attend each class and participate in discussion by asking thoughtful questions and communicating their opinions, ideas, and understanding on a topic clearly to everyone in the class.

Schedule

Week 1 : Introduction

Themes covered:

  • Mental Models, Conceptual Models, System Image,
  • Human-Centered Machine Learning as a broad topic of research
  • Common mistakes in machine learning

Discussion Prompt :

  • What are the different ways that people envision using AI and how does machine learning fit into these ideas?
  • How do you see users and their mental models relate to machine learning and AI?
  • What are the common pitfalls of machine learning that Domingos' presents and how might a "user" address or remedy these challenges?

Readings :

Week 2 : Interactive Machine Learning 1 : Introduction

Themes Covered:

  • Interactive ML
  • Machine Teaching

Discussion Prompt:

  • What are some of the trade-offs that are made to make an ML model suitable for interactivity?
  • Describe some of the user experiences for data labeling.

Readings:

Week 3 : Interactive Machine Learning 2 : Active Learning

Themes covered:

  • Active Learning
  • Machine Teaching
  • Interactive Feature Selection / Model Selection

Discussion Prompt :

  • What are some kinds of model feedback and how do they affect user interaction? Why does feedback affect user interaction?
  • What are some ways in which current machine learning algorithms and methods have been modified for suitability in IML?
  • What are some of the drawbacks to Active Learning and why might (or might not) they be addressed in the UI/UX design?

Readings :

Week 4 : Interactive Machine Learning 2: Reinforcement Learning

Themes covered:

  • Programming by Demonstration / Learning By Example / Apprenticeship Learning
  • reinforcement learning
  • inverse reinforcement learning

Discussion Prompts:

  • From a user perspective, what makes apprenticeship learning so different from active learning?
  • Describe the state and action space represent in each of these papers.

Readings:

Week 5 : Model Personalization

Themes:

  • Semi-supervised learning
  • Community Similarity
  • incremental and online learning

Discussion Prompt:

  • What is incremental learning and what is online learning? Why do these areas of ML research aid the model personalization process?
  • How might we decide to use either active learning, semi-supervised learning, or community similarity approaches for the purpose of model personalization? To help reason about this choose a specific application domain (many of these papers chose activity recognition) to reason about the decision.

Readings:

Week 6 : Model Transparency

Themes:

Discussion Prompts:

  • What roles do interpretability and visualization play in helping a model to be truly "transparent"?
  • How do multiple "views" help users to better understand data and models based on it.

Readings:

Week 7 : Algorithmic Bias, Accountability, and Auditing

Themes:

  • Algorithmic Bias,
  • Algorithmic Accountability
  • Algorithmic Auditing

Discussion Prompts:

  • What is the difference between "user-centered" and "human-centered" and which best describes each of the papers for this week?
  • What can we "audit" an algorithm for?
  • What is a good description of how biases can manifest themselves in the implementation or design of an automated or algorithm driven system.

Readings:

Week 8 : Crowd ML 1

Themes:

  • Community-guided Learning
  • Citizen Science

Discussion Prompts:

  • What are benefits to the user or the model that we derive from combining information from more than one user?
    • What are the challenges?
  • What (if anything) do the users get in exchange for their effort or privacy violation when giving up information?
  • If some subset of the users in a crowd are providing incorrect or inadequate information (either because of malicious intent or misinformation), what kind of effect will this have on the system's ability to achieve it's purpose? First, think through the purpose of the system, then think through likely ways information could be incorrect or inadequate considering user intent, then think through the effect this will have on the model, finally how will the ultimate model effect the initial purpose.

Readings:

Week 9 : Crowd ML 2

Themes:

  • Crowd Work
  • Grounded Theory
  • Model Diagnostics

Discussion Prompts:

  • Why is it important to make turk work salient in the automated systems that depend on crowd workers?
  • How can human knowledge compliment automated knowledge discovery and vice versa.
  • What are important design decisions that should be made when eliciting high quality information from the crowd for the purpose of training a model.

Readings:

Week 10 : Design Principles for Intelligent User Interfaces

Themes:

  • Keyphrases
  • User Feedback
  • Explanatory Debugging

Discussion Prompts:

  • What is an intelligent user interface and what effect does user feedback have on the system behavior?
  • How could keyphrases be made practical in applications of text mining? Be explicit about what this work process might look like.
  • How can the research process used to create design principles for explanatory debugging be applied to other domains we have discussed throughout this course?

Readings:

Additional Resources:

Looking for inspiration

Other Labs and Courses

Things to know when designing a labeling task

Other good things to know about