bnmuriithi/human-centered-machine-learning

Schedule and Syllabus for Human-Centered Machine learning.

Human-Centered Machine Learning (Spring 2021)

Course staff

Insturctor: Chenhao Tan contact
Office hours: 2:30-3:30pm on Tuesday or by appointment

Logistics

• Location and time: Tuesday and Thursday 1:00-2:20pm on Zoom
• Syllabus (Must READ if you are taking the course)
• Canvas
• Zoom link (available on canvas)
• Ed (available on canvas)

Schedule

• Week 1: Introduction

  • Mar 30, Introduction
  • Apr 1, Ask not what AI can do, but what AI should do: Towards a Framework of Task Delegability. Brian Lubars and Chenhao Tan. NeurIPS 2019.
    Additional reading:
    • Human-centered Machine Learning: a Machine-in-the-loop Approach, Chenhao Tan.
    • Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making. Gabriel Lima, Nina Grgić-Hlača, Meeyoung Cha. CHI 2021.

• Week 2: You are not so Smart

  • Apr 6, Human Decisions and Machine Predictions. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan. Quarterly Journal of Economics, 2018.
  • Apr 8, Telling More Than We Can Know: Verbal Reports on Mental Processes (pdf is available on CANVAS). Richard E. Nisbett and Timothy DeCamp Wilson. Psychological Review. 1977.
    Additional reading:
    • Judgment under Uncertainty: Heuristics and Biases. Amos Tversky and Daniel Kahneman, Science, 1974.
    • Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. Marianne Bertrand and Sendhil Mullainathan. American Economic Review. 2004.
    • An algorithmic approach to reducing unexplained pain disparities in underserved populations. Emma Pierson, David M. Cutler, Jure Leskovec, Sendhil Mullainathan, and Ziad Obermeyer. Nature Medicine. 2021.
    • Assessing Human Error Against a Benchmark of Perfection. Ashton Anderson, Jon Kleinberg, Sendhil Mullainathan. KDD 2016.
    • Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey, Journal of Experimental Psychology: General, 2014.
    • Podcast: You are not so smart
    • The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter. Chenhao Tan, Lillian Lee, Bo Pang. ACL 2014.
    • Thinking, Fast and Slow. Daniel Kahneman. 2011.
      First proposal due on Apr 9

• Week 3: Machine Bias

  • Apr 13, Dissecting racial bias in an algorithm used to manage the health of populations. Ziad Obermeyer, Brian Powers, Christine Vogeli, Sendhil Mullainathan. Science. 2019.
  • Apr 15, Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Joy Buolamwini and Timnit Gebru. FAccT 2018.
    Additional reading:
    • Language (Technology) is Power: A Critical Survey of "Bias" in NLP. Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach. 2020.
    • Discrimination in Online Ad Delivery. Latanya Sweeney. Communications of the ACM, 2013.
    • Racial disparities in automated speech recognition. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. PNAS. 2020.
    • The Risk of Racial Bias in Hate Speech Detection. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A. Smith. ACL 2019.
    • Gender bias in coreference resolution: Evaluation and debiasing methods. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. NAACL 2018.
    • Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP 2017.
    • Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai. NeurIPS 2016.

• Week 4: Accountability

  • Apr 20, Roles for Computing in Social Change. Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David G. Robinson. FAccT 2020.
  • Apr 22, Model Cards for Model Reporting. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. FAccT 2019.
    Additional reading:
    • Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Inioluwa Deborah Raji and Joy Buolamwini. AIES 2019.
    • Datasheets for datasets. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. 2018.
    • European Union regulations on algorithmic decision-making and a "right to explanation". Bryce Goodman and Seth Flaxman.
    • Equality of opportunity in supervised learning. Moritz Hardt, Eric Price, Nathan Srebro. NeurIPS 2016.
    • Inherent Trade-Offs in the Fair Determination of Risk Scores. Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan. ITCS 2017.
    • Algorithmic decision making and the cost of fairness. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq. KDD 2017.
    • Fairness and Abstraction in Sociotechnical Systems. Andrew D. Selbst, danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, Janet Vertesi. FAT* 2019.
      Second proposal due on Apr 23.

• Week 5: Machine-in-the-loop Interactions

  • Apr 27, Guidelines for Human-AI Interaction. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. CHI 2019.
  • Apr 29, Towards A Rigorous Science of Interpretable Machine Learning. Finale Doshi-Velez and Been Kim.
    Additional reading:
    • Principles of Mixed-Initiative User Interfaces. Eric Horvitz. In Proceedings of CHI, 1999.
    • Questioning the AI: informing design practices for explainable AI user experiences. Q. Vera Liao, Daniel Gruen, Sarah Miller. CHI 2020.
    • Human-centered artificial intelligence: Reliable, safe & trustworthy. Ben Shneiderman. International Journal of Human-Computer Interaction. 2020.
    • Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, Mohan Kankanhalli. CHI 2018.
    • Interpretable machine learning: definitions, methods, and applications. W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu. PNAS 2019.
    • Natural Language Translation at the Intersection of AI and HCI. Spence Gree, Jeffrey Heer, Christopher D. Manning. Queue, 2015.
    • A Review of User Interface Design for Interactive Machine Learning. John J. Dudley and Per Ola Kristensson. ACM Transactions on Interactive Intelligent Systems. 2018.
    • Beyond binary choices: Integrating individual and social creativity. Gerhard Fischer, Elisa Giaccardi, Hal Eden, Masanori Sugimoto, Yunwen Ye. International Journal of Human-Computer Studies, 2005.
    • The Mythos of Model Interpretability, Zachary C. Lipton.
    • A Human-Centered Agenda for Intelligible Machine Learning. Jennifer Wortman Vaughan and Hanna Wallach.
    • Who is the “Human” in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media. Stevie Chancellor, Eric P.S Baumer, and Munmun De Choudhury. CSCW 2019.
    • Human Centered Systems in the Perspective of Organizational and Social Informatics. Rob Kling and Leigh Star. 1997.
      Team formation due on Apr 30.

• Week 6: Feature-based Explanations

  • May 4, Why should I trust you?: Explaining the Predictions of Any Classifier. Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. KDD 2016.
  • May 6, Axiomatic attribution for deep networks. Mukund Sundararajan, Ankur Taly, Qiqi Yan. ICML 2017.
    Additional reading:
    • A Unified Approach to Interpreting Model Predictions. Scott Lundberg, Su-In Lee. NeurIPS 2017.
    • Attention is not explanations Sarthak Jain, Byron C. Wallace. NAACL 2019.
    • Attention is not not explanations. Sarah Wiegreffe, Yuval Pinter. EMNLP 2019.
    • Is Attention Interpretable?. Sofia Serrano, Noah A. Smith. ACL 2019.
    • Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification. Vivian Lai, Jon Z. Cai, Chenhao Tan. EMNLP 2019.
    • DeepXplore: Automated Whitebox Testing of Deep Learning Systems, Pei, Cao, Yang and Jana. In Proceedings of SOSP, 2017.
    • Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola. In Proceedings of EMNLP, 2016.
    • Learning Explanatory Rules from Noisy Data. Richard Evans, Edward Grefenstette. Journal of Artificial Intelligence Research, 2018.
    • Grad-cam: Visual explanations from deep networks via gradient-based localization. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra. ICCV 2017.
    • Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness? Alon Jacovi, Yoav Goldberg. ACL 2020.
    • Evaluating and Characterizing Human Rationales. Samuel Carton, Anirudh Rathore, Chenhao Tan. EMNLP 2020.
    • Network Dissection: Quantifying Interpretability of Deep Visual Representations. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. In Proceedings of CVPR 2017.

• Week 7: Example-based explanations

  • May 11, Deep Weighted Averaging Classifiers. Dallas Card, Michael Zhang, Noah A. Smith. FAccT 2019.
  • May 13, Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Sandra Wachter, Brent Mittelstadt, Chris Russell. 2017.
    Additional reading:
    • Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Been Kim, Rajiv Khanna, Oluwasanmi Koyejo. NeurIPS 2016.
    • Case-based explanation of non-case-based learning methods. Rich Caruana, Hooshang Kangarloo, John David N. Dionisio, Usha Sinha, David Johnson. AMIA 1999.
    • How Case Based Reasoning Explained Neural Networks: An XAI Survey of Post-Hoc Explanation-by-Example in ANN-CBR Twins. Mark T Keane, Eoin M Kenny. ICCBR 2019.
    • Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. Nicolas Papernot, Patrick McDaniel.
    • Interactive and Interpretable Machine Learning Models for Human Machine Collaboration, Been Kim, PhD thesis.
    • Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. Ramaravind Kommiya Mothilal, Amit Sharma, Chenhao Tan. FAT* 2020.
    • The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons. Solon Barocas, Andrew D. Selbst, Manish Raghavan. FAT* 2020.
    • Efficient Search for Diverse Coherent Explanations. Chris Russell. FAT* 2019.
    • This Looks Like That: Deep Learning for Interpretable Image Recognition. Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, Cynthia Rudin. NeurIPS 2019.

• Week 8: Human-AI interaction --- Decision Making

  • May 18, On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. Vivian Lai, Chenhao Tan. FAccT 2019.
  • May 20, The Principles and Limits of Algorithm-in-the-Loop Decision Making. Ben Green, Yiling Chen. CSCW 2019.
    Additional reading:
    • The limits of human predictions of recidivism. Zhiyuan “Jerry” Lin, Jongbin Jung, Sharad Goel, Jennifer Skeem. Science Advances 2020.
    • Human Evaluation of Models Built for Interpretability. Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman and Finale Doshi-Velez. HCOMP 2019.
    • Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. IUI 2020.
    • Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. Gagan Bansal*, Tongshuang Wu*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld. CHI 2021.
    • Prediction Policy Problems. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, Ziad Obermeyer. American Economic Review, 2015.
    • The Tyranny of Data? The Bright and Dark Sides of Data-Driven Decision-Making for Social Good. Bruno Lepri, Jacopo Staiano, David Sangokoya, Emmanuel Letouzé, and Nuria Oliver. Transparent Data Mining for Big and Small Data, 2017.
    • Predicting the knowledge–recklessness distinction in the human brain. Iris Vilares, Michael J. Wesley, Woo-Young Ahn, Richard J. Bonnie, Morris Hoffman, Owen D. Jones, Stephen J. Morse, Gideon Yaffe, Terry Lohrenz, and P. Read Montague. PNAS, 2016.

• Week 9: Human-AI interaction --- Trust

  • May 25, Trust in automation: Designing for appropriate reliance. John Lee and Katrina See. Human factors, 2004.
  • May 27, Impact of a deep learning assistant on the histopathologic classification of liver cancer. Amirhossein Kiani, Bora Uyumazturk, Pranav Rajpurkar, Alex Wang, Rebecca Gao, Erik Jones, Yifan Yu, Curtis P. Langlotz, Robyn L. Ball, Thomas J. Montine, Brock A. Martin, Gerald J. Berry, Michael G. Ozawa, Florette K. Hazard, Ryanne A. Brown, Simon B. Chen, Mona Wood, Libby S. Allard, Lourdes Ylagan, Andrew Y. Ng & Jeanne Shen. npj Digital Medicine, 2020.
    Additional reading:
    • To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Zana Buçinca, Maja Barbara Malaya, Krzysztof Z. Gajos. CSCW 2021.
    • Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. Alon Jacovi, Ana Marasović, Tim Miller, Yoav Goldberg. FAccT 2021.
    • Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. CHI 2019.
    • Understanding the Effect of Accuracy on Trust in Machine Learning Models. Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. CHI 2019.

• Week 10: Final Project Presentation

  • TBD, presentations (could be different from normal class time).
    Final project report due on Jun 1.