gouliev/human-centered-machine-learning

Schedule and Syllabus for Human-Centered Machine learning.

Human-Centered Machine Learning (Spring 2022)

Course staff

Insturctor: Chenhao Tan contact
Office hours: 2:00-3:00pm on Tuesdays or by appointment

Logistics

• Location and time: Tuesday and Thursday 3:30-4:50pm at Kersten Physics Teaching Center 105
• Syllabus (Must READ if you are taking the course)
• Canvas
• Ed (available on canvas)

Schedule

• Week 1: Introduction

  • Mar 29, Introduction
  • Mar 31, Ask not what AI can do, but what AI should do: Towards a Framework of Task Delegability. Brian Lubars and Chenhao Tan. NeurIPS 2019.
    Additional reading:
    • Human-centered Machine Learning: a Machine-in-the-loop Approach, Chenhao Tan.
    • Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making. Gabriel Lima, Nina Grgić-Hlača, Meeyoung Cha. CHI 2021.

• Week 2: Human + Machine

  • Apr 5, Human Decisions and Machine Predictions. Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, Sendhil Mullainathan. Quarterly Journal of Economics, 2018.
  • Apr 7, Roles for Computing in Social Change. Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, David G. Robinson. FAccT 2020.
    Additional reading:
    • Judgment under Uncertainty: Heuristics and Biases. Amos Tversky and Daniel Kahneman, Science, 1974.
    • Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. Marianne Bertrand and Sendhil Mullainathan. American Economic Review. 2004.
    • An algorithmic approach to reducing unexplained pain disparities in underserved populations. Emma Pierson, David M. Cutler, Jure Leskovec, Sendhil Mullainathan, and Ziad Obermeyer. Nature Medicine. 2021.
    • Assessing Human Error Against a Benchmark of Perfection. Ashton Anderson, Jon Kleinberg, Sendhil Mullainathan. KDD 2016.
    • Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err. Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey, Journal of Experimental Psychology: General, 2014.
    • Podcast: You are not so smart
    • The effect of wording on message propagation: Topic- and author-controlled natural experiments on Twitter. Chenhao Tan, Lillian Lee, Bo Pang. ACL 2014.
    • Thinking, Fast and Slow. Daniel Kahneman. 2011.
      First proposal due on Apr 8

• Week 3: Machine Bias

  • Apr 12, Dissecting racial bias in an algorithm used to manage the health of populations. Ziad Obermeyer, Brian Powers, Christine Vogeli, Sendhil Mullainathan. Science. 2019.
  • Apr 14, Language (Technology) is Power: A Critical Survey of "Bias" in NLP. Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach. 2020. Additional reading:
    • Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Joy Buolamwini and Timnit Gebru. FAccT 2018.
    • Discrimination in Online Ad Delivery. Latanya Sweeney. Communications of the ACM, 2013.
    • Racial disparities in automated speech recognition. Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. PNAS. 2020.
    • The Risk of Racial Bias in Hate Speech Detection. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A. Smith. ACL 2019.
    • Gender bias in coreference resolution: Evaluation and debiasing methods. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. NAACL 2018.
    • Men also like shopping: Reducing gender bias amplification using corpus-level constraints. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, Kai-Wei Chang. EMNLP 2017.
    • Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai. NeurIPS 2016.

• Week 4: Accountability

  • Apr 19, Mitigating bias in algorithmic hiring: evaluating claims and practices. Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy. FAccT 2020.
  • Apr 21, Model Cards for Model Reporting. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, Timnit Gebru. FAccT 2019.
    Additional reading:
    • Building and Auditing Fair Algorithms: A Case Study in Candidate Screening. Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, Frida Polli. FAccT 2021.
    • Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. Inioluwa Deborah Raji and Joy Buolamwini. AIES 2019.
    • Datasheets for datasets. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. 2018.
    • European Union regulations on algorithmic decision-making and a "right to explanation". Bryce Goodman and Seth Flaxman.
    • Equality of opportunity in supervised learning. Moritz Hardt, Eric Price, Nathan Srebro. NeurIPS 2016.
    • Inherent Trade-Offs in the Fair Determination of Risk Scores. Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan. ITCS 2017.
    • Algorithmic decision making and the cost of fairness. Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Huq. KDD 2017.
    • Fairness and Abstraction in Sociotechnical Systems. Andrew D. Selbst, danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, Janet Vertesi. FAT* 2019.
      Second proposal due on Apr 22.

• Week 5: Machine-in-the-loop Interactions

  • Apr 26, Guidelines for Human-AI Interaction. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. CHI 2019.
  • Apr 28, Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies. Vivian Lai, Chacha Chen, Q. Vera Liao, Alison Smith-Renner, Chenhao Tan. Additional reading:
    • Towards A Rigorous Science of Interpretable Machine Learning. Finale Doshi-Velez and Been Kim.
    • Principles of Mixed-Initiative User Interfaces. Eric Horvitz. In Proceedings of CHI, 1999.
    • Questioning the AI: informing design practices for explainable AI user experiences. Q. Vera Liao, Daniel Gruen, Sarah Miller. CHI 2020.
    • Human-centered artificial intelligence: Reliable, safe & trustworthy. Ben Shneiderman. International Journal of Human-Computer Interaction. 2020.
    • Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, Mohan Kankanhalli. CHI 2018.
    • Interpretable machine learning: definitions, methods, and applications. W. James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, Bin Yu. PNAS 2019.
    • Natural Language Translation at the Intersection of AI and HCI. Spence Gree, Jeffrey Heer, Christopher D. Manning. Queue, 2015.
    • A Review of User Interface Design for Interactive Machine Learning. John J. Dudley and Per Ola Kristensson. ACM Transactions on Interactive Intelligent Systems. 2018.
    • Beyond binary choices: Integrating individual and social creativity. Gerhard Fischer, Elisa Giaccardi, Hal Eden, Masanori Sugimoto, Yunwen Ye. International Journal of Human-Computer Studies, 2005.
    • The Mythos of Model Interpretability, Zachary C. Lipton.
    • A Human-Centered Agenda for Intelligible Machine Learning. Jennifer Wortman Vaughan and Hanna Wallach.
    • Who is the “Human” in Human-Centered Machine Learning: The Case of Predicting Mental Health from Social Media. Stevie Chancellor, Eric P.S Baumer, and Munmun De Choudhury. CSCW 2019.
    • Human Centered Systems in the Perspective of Organizational and Social Informatics. Rob Kling and Leigh Star. 1997.
      Team formation due on Apr 29.

• Week 6: Feature-based Explanations

  • May 3, Why should I trust you?: Explaining the Predictions of Any Classifier. Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. KDD 2016.
  • May 5, Aligning Faithful Interpretations with their Social Attribution. Alon Jacovi and Yoav Goldberg. TACL 2021. Additional reading:
    • How can I choose an explainer?: An Application-grounded Evaluation of Post-hoc Explanations. Sérgio Jesus, Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, João Gama. FAccT 2021.
    • A Unified Approach to Interpreting Model Predictions. Scott Lundberg, Su-In Lee. NeurIPS 2017.
    • Attention is not explanations Sarthak Jain, Byron C. Wallace. NAACL 2019.
    • Attention is not not explanations. Sarah Wiegreffe, Yuval Pinter. EMNLP 2019.
    • Is Attention Interpretable?. Sofia Serrano, Noah A. Smith. ACL 2019.
    • Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification. Vivian Lai, Jon Z. Cai, Chenhao Tan. EMNLP 2019.
    • DeepXplore: Automated Whitebox Testing of Deep Learning Systems, Pei, Cao, Yang and Jana. In Proceedings of SOSP, 2017.
    • Rationalizing Neural Predictions, Tao Lei, Regina Barzilay and Tommi Jaakkola. In Proceedings of EMNLP, 2016.
    • Learning Explanatory Rules from Noisy Data. Richard Evans, Edward Grefenstette. Journal of Artificial Intelligence Research, 2018.
    • Grad-cam: Visual explanations from deep networks via gradient-based localization. Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra. ICCV 2017.
    • Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness? Alon Jacovi, Yoav Goldberg. ACL 2020.
    • Evaluating and Characterizing Human Rationales. Samuel Carton, Anirudh Rathore, Chenhao Tan. EMNLP 2020.
    • Network Dissection: Quantifying Interpretability of Deep Visual Representations. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba. In Proceedings of CVPR 2017.
    • Axiomatic attribution for deep networks. Mukund Sundararajan, Ankur Taly, Qiqi Yan. ICML 2017.

• Week 7: Example-based explanations

  • May 10, This Looks Like That: Deep Learning for Interpretable Image Recognition. Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, Cynthia Rudin. NeurIPS 2019.
  • May 12, Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning. Yongchan Kwon, James Zou. AISTATS 2022. Additional reading:
    • The Use and Misuse of Counterfactuals in Ethical Machine Learning. Atoosa Kasirzadeh, Andrew Smart. FAccT 2021.
    • Data Shapley: Equitable Data Valuation for Machine Learning. Amirata Ghorbani, James Zou. ICML 2019.
    • Understanding Black-box Predictions via Influence Functions. Pang Wei Koh, Percy Liang. ICML 2017.
    • Examples are not Enough, Learn to Criticize! Criticism for Interpretability. Been Kim, Rajiv Khanna, Oluwasanmi Koyejo. NeurIPS 2016.
    • Case-based explanation of non-case-based learning methods. Rich Caruana, Hooshang Kangarloo, John David N. Dionisio, Usha Sinha, David Johnson. AMIA 1999.
    • How Case Based Reasoning Explained Neural Networks: An XAI Survey of Post-Hoc Explanation-by-Example in ANN-CBR Twins. Mark T Keane, Eoin M Kenny. ICCBR 2019.
    • Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. Nicolas Papernot, Patrick McDaniel.
    • Interactive and Interpretable Machine Learning Models for Human Machine Collaboration, Been Kim, PhD thesis.
    • Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations. Ramaravind Kommiya Mothilal, Amit Sharma, Chenhao Tan. FAccT 2020.
    • The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons. Solon Barocas, Andrew D. Selbst, Manish Raghavan. FAT* 2020.
    • Efficient Search for Diverse Coherent Explanations. Chris Russell. FAccT 2019.
    • Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Sandra Wachter, Brent Mittelstadt, Chris Russell. 2017.
    • Data valuation for medical imaging using Shapley value and application to a large-scale chest X-ray dataset. Siyi Tang, Amirata Ghorbani, R. Yamashita, S. Rehman, Jared Dunnmon, James Zou, Daniel Rubin. Scientific Reports (2021).

• Week 8: Human-AI interaction --- Decision Making

  • May 17, On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. Vivian Lai, Chenhao Tan. FAccT 2019.
  • May 19, The Principles and Limits of Algorithm-in-the-Loop Decision Making. Ben Green, Yiling Chen. CSCW 2019.
    Additional reading:
    • The limits of human predictions of recidivism. Zhiyuan “Jerry” Lin, Jongbin Jung, Sharad Goel, Jennifer Skeem. Science Advances 2020.
    • Human Evaluation of Models Built for Interpretability. Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Samuel Gershman and Finale Doshi-Velez. HCOMP 2019.
    • Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. IUI 2020.
    • Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. Gagan Bansal*, Tongshuang Wu*, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, Daniel S. Weld. CHI 2021.
    • Prediction Policy Problems. Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, Ziad Obermeyer. American Economic Review, 2015.
    • The Tyranny of Data? The Bright and Dark Sides of Data-Driven Decision-Making for Social Good. Bruno Lepri, Jacopo Staiano, David Sangokoya, Emmanuel Letouzé, and Nuria Oliver. Transparent Data Mining for Big and Small Data, 2017.
    • Predicting the knowledge–recklessness distinction in the human brain. Iris Vilares, Michael J. Wesley, Woo-Young Ahn, Richard J. Bonnie, Morris Hoffman, Owen D. Jones, Stephen J. Morse, Gideon Yaffe, Terry Lohrenz, and P. Read Montague. PNAS, 2016.

• Week 9: Human-AI interaction --- Trust

  • May 24, To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making. Zana Buçinca, Maja Barbara Malaya, Krzysztof Z. Gajos. CSCW 2021.
  • May 26, Teaching Humans When To Defer to a Classifier via Exemplars. Hussein Mozannar, Arvind Satyanarayan, David Sontag. AAAI 2022. Additional reading:
    • Impact of a deep learning assistant on the histopathologic classification of liver cancer. Amirhossein Kiani, Bora Uyumazturk, Pranav Rajpurkar, Alex Wang, Rebecca Gao, Erik Jones, Yifan Yu, Curtis P. Langlotz, Robyn L. Ball, Thomas J. Montine, Brock A. Martin, Gerald J. Berry, Michael G. Ozawa, Florette K. Hazard, Ryanne A. Brown, Simon B. Chen, Mona Wood, Libby S. Allard, Lourdes Ylagan, Andrew Y. Ng & Jeanne Shen. npj Digital Medicine, 2020.
    • Trust in automation: Designing for appropriate reliance. John Lee and Katrina See. Human factors, 2004.
    • Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI. Alon Jacovi, Ana Marasović, Tim Miller, Yoav Goldberg. FAccT 2021.
    • Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. Johannes Kunkel, Tim Donkers, Lisa Michael, Catalin-Mihai Barbu, and Jürgen Ziegler. CHI 2019.
    • Understanding the Effect of Accuracy on Trust in Machine Learning Models. Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. CHI 2019.

• Week 10: Final Project Presentation

  • TBD, presentations (could be different from normal class time).
    Final project report due on May 31.