Machine learning (ML) models are increasingly being employed to make highly consequential decisions pertaining to employment, bail, parole, and lending. While such models can learn from large amounts of data and are often very scalable, their applicability is limited by certain safety challenges. A key challenge is identifying and correcting systematic patterns of mistakes made by ML models before deploying them in the real world.

The goal of this workshop, held at the 2019 International Conference on Learning Representations (ICLR), is to bring together researchers and practitioners with different perspectives on debugging ML models.


Aleksander Madry
Aleksander Madry
Cynthia Rudin
Cynthia Rudin
Duke University
Dan Moldovan
Dan Moldovan
Deborah Raji
Deborah Raji
University of Toronto
Osbert Bastani
Osbert Bastani
University of Pennsylvania
Sameer Singh
Sameer Singh
UC Irvine
Suchi Saria
Suchi Saria
Johns Hopkins University


See here for a printable version.

Time Event
9.50 Opening remarks [video]
  Session Chair: Julius Adebayo (MIT)
10.00 Invited talkAleksander Madry (MIT): A New Perspective on Adversarial Perturbations [video]
10:30 Contributed talk (Best Research Paper Award)Simon Kornblith (Google): Similarity of Neural Network Representations Revisited [video]
10.40 Contributed talk (Best Demo Award)Besmira Nushi (Microsoft Research): Error terrain analysis for machine learning: Tool and visualizations [video]
10.50 Coffee break
  Session Chair: Julius Adebayo (MIT)
11.10 Invited talkOsbert Bastani (University of Pennsylvania): Verifiable Reinforcement Learning via Policy Extraction [video]
11:40 Contributed talk (Best Student Research Paper Award)Daniel Kang (Stanford): Debugging Machine Learning via Model Assertions [video]
11:50 Contributed talkBenjamin Link (Indeed): Improving Jobseeker-Employer Match Models at Indeed Through Process, Visualization, and Exploration [video]
12.00 Break
  Session Chair: Sarah Tan (Cornell University / UCSF)
12.10 Invited talkSameer Singh (UC Irvine): Discovering Natural Bugs Using Adversarial Data Perturbations [video]
12.40 Invited talkDeborah Raji (University of Toronto): “Debugging” Discriminatory ML Systems [video]
1.00 Contributed talk (Best Applied Paper Award)Tomer Arnon and Christopher Lazarus: NeuralVerification.jl: Algorithms for Verifying Deep Neural Networks [video]
1.10 Lunch
2.30 Break
  Session Chair: D Sculley (Google)
3.20 Welcome back remarks
3.30 Invited talkSuchi Saria (Johns Hopkins University): Safe and Reliable Machine Learning: Preventing and Identifying Failures [video]
4.00 Invited talkDan Moldovan (Google): Better Code for Less Debugging with AutoGraph [video]
4.20 Posters & Demos & Coffee break
Accepted posters   Accepted demos
  Session Chair: Himabindu Lakkaraju (Harvard University)
5.20 Contributed position paperMichela Paganini (Facebook): The Scientific Method in the Science of Machine Learning [video]
5.30 Invited opinion pieceCynthia Rudin (Duke University): Don’t debug your black box, replace it [video]
6.00 Q&A and panel with all invited speakers – “The Future of ML Debugging” [video]
Moderator: Himabindu Lakkaraju (Harvard University)
Panelists: Aleksander Madry, Cynthia Rudin, Dan Moldovan, Deborah Raji, Osbert Bastani, Sameer Singh, Suchi Saria
6.25 Closing remarks


Call for submissions (deadline has passed)


Call for submissions (deadline has passed)


See a list of references.


Himabindu Lakkaraju
Himabindu Lakkaraju
Harvard University
Sarah Tan
Sarah Tan
Cornell University / UCSF
Julius Adebayo
Julius Adebayo
Jacob Steinhardt
Jacob Steinhardt
Open Philanthropy Project / OpenAI
D Sculley
D Sculley
Rich Caruana
Rich Caruana
Microsoft Research

Contact Us

Email debugging.ml@gmail.com any questions.


OpenAI logo
Google logo

Program Committee

Samira Abnar (University of Amsterdam) Lezhi Li (Uber)
David Alvarez Melis (MIT) Anqi Liu (Caltech)
Forough Arabshahi (Carnegie Mellon University) Yin Lou (Ant Financial)
Kamyar Azzizzadenesheli (UC Irvine) David Madras (University of Toronto / Vector Institute)
Gagan Bansal (University of Washington) Sara Magliacane (IBM Research)
Osbert Bastani (University of Pennsylvania) Momin Malik (Berkman Klein Center)
Joost Bastings (University of Amsterdam) Matthew Mcdermott (MIT)
Andrew Beam (Harvard University) Smitha Milli (UC Berkeley)
Kush Bhatia (UC Berkeley) Shira Mitchell ()
Umang Bhatt (Carnegie Mellon University) Tristan Naumann (Microsoft Research)
Cristian Canton (Facebook) Besmira Nushi (Microsoft Research)
Arthur Choi (UCLA) Saswat Padhi (UCLA)
Grzegorz Chrupala (Tilburg University) Emma Pierson (Stanford University)
Sam Corbett-Davies (Facebook) Forough Poursabzi-Sangdeh (Microsoft Research)
Amit Dhurandhar (IBM Research) Manish Raghavan (Cornell University)
Samuel Finlayson (Harvard Medical School, MIT) Ramya Ramakrishnan (MIT)
Tian Gao (IBM Research) Alexander Ratner (Stanford University)
Efstathios Gennatas (UCSF) Andrew Ross (Harvard University)
Siongthye Goh (Singapore Management University) Shibani Santurkar (MIT)
Albert Gordo (Facebook) Prasanna Sattigeri (IBM Research)
Ben Green (Harvard University) Peter Schulam (Johns Hopkins University)
Jayesh Gupta (Stanford University) Ravi Shroff (NYU)
Satoshi Hara (Osaka University) Camelia Simoiu (Stanford University)
Tatsunori Hashimoto (MIT) Sameer Singh (UC Irvine)
He He (NYU) Alison Smith-Renner (University of Maryland)
Fred Hohman (Georgia Institute of Technology) Jina Suh (Microsoft Research)
Lily Hu (Harvard University) Adith Swaminathan (Microsoft Research)
Xiaowei Huang (University of Liverpool) Michael Tsang (University of Southern California)
Yannet Interian (University of San Francisco) Dimitris Tsipras (MIT)
Saumya Jetley (University of Oxford) Berk Ustun (Harvard University)
Shalmali Joshi (Vector Institute) Gilmer Valdes (UCSF)
Yannis Kalantidis (Facebook) Paroma Varma (Stanford University)
Ece Kamar (Microsoft Research) Kush Varshney (IBM Research)
Madian Khabsa (Facebook) Fulton Wang (Sandia National Labs)
Heidy Khlaaf (Adelard) Yang Wang (Uber)
Pang Wei Koh (Stanford University) Fanny Yang (ETH Zurich)
Josua Krause (Accern) Jason Yosinski (Uber)
Ram Kumar (Microsoft / Berkman Klein Center) Muhammad Bilal Zafar (Bosch Center for Artificial Intelligence)
Isaac Lage (Harvard University) Xuezhou Zhang (University of Wisconsin-Madison)
Finnian Lattimore (Australian National University) Xin Zhang (MIT)
Marco Tulio Ribeiro (Microsoft Research)