Machine learning (ML) models are increasingly being employed to make highly consequential decisions pertaining to employment, bail, parole, and lending. While such models can learn from large amounts of data and are often very scalable, their applicability is limited by certain safety challenges. A key challenge is identifying and correcting systematic patterns of mistakes made by ML models before deploying them in the real world.

The goal of this workshop, held at the 2019 International Conference on Learning Representations (ICLR), is to bring together researchers and practitioners with different perspectives on debugging ML models.

Invited Speakers


See here for a printable version.

Time Event
9.50 - 10:00 Opening Remarks
10.00 - 10.30 Invited Talk: Aleksander Madry (MIT).
10:30 - 10:40 Contributed Talk: Similarity of Neural Network Representations Revisited.
Simon Kornblith, Mohammad Norouzi, Honglak Lee and Geoffrey Hinton (Google).
10.40 - 10:50 Contributed Talk: Error terrain analysis for machine learning: Tool and visualizations.
Rick Barraza, Russell Eames, Yan Esteve Balducci, Josh Hinds, Scott Hoogerwerf, Eric Horvitz, Ece Kamar, Jacquelyn Krones, Josh Lovejoy, Parham Mohadjer, Ben Noah and Besmira Nushi (Microsoft).
10.50 - 11.10 Coffee Break
11.10 - 11.40 Invited Talk: Osbert Bastani (University of Pennsylvania).
10:40 - 10:50 Contributed Talk: Debugging Machine Learning via Model Assertions.
Daniel Kang, Deepti Raghavan, Peter Bailis, Matei Zaharia (Stanford).
10:50 - 12:00 Contributed Talk: Improving jobseeker-employer match models at Indeed through process, visualization, and exploration.
Benjamin Link, Eric Lawrence, & Rosemarie Scott (Indeed).
12.00 - 12.10 Break
11.10 - 11.40 Invited Talk: Sameer Singh (University of California Irvine).
12.40 - 1.00 Invited Talk: Deborah Raji (University of Toronto).
1.00 - 1:10 Contributed Talk: NeuralVerification.jl: Algorithms for Verifying Deep Neural Networks.
Changliu Liu (CMU), Tomer Arnon, Christopher Lazarus and Mykel Kochenderfer (Stanford).
1.10 - 2.20 Lunch
2.30 - 3.20 Break
3.20 - 3.30 Welcome back remarks
3.30 - 4.00 Invited Talk: Suchi Saria (Johns Hopkins University).
4.00 - 4.20 Invited Talk: Dan Moldovan (Google).
4.20 - 4.20 Posters & Demos & Coffee Break
5.20 - 5.30 Contributed Talk: The Scientific Method in the Science of Machine Learning.
Jessica Zosa Forde (Project Jupyter), Michela Paganini (Facebook).
5.30 - 6.00 Invited Talk: Cynthia Rudin (Duke University).
6.00 - 6.25 Q&A/Panel with all invited speakers: “The Future of ML Debugging.”
Moderator: Rich Caruana (Microsoft Research).
Panelists: Aleksander Madry, Cynthia Rudin, Dan Moldovan, Deborah Raji, Osbert Bastani, Sameer Singh, Suchi Saria
6.25 - 6.30 Closing Remarks.

Contributed Posters (Research Track)

Call for submissions (deadline has passed)

Contributed Demos (Debugging-in-Practice Track)

Call for submissions (deadline has passed)

Workshop Topics

See a list of references.


Contact Us

Email debugging.ml@gmail.com any questions.


OpenAI logo

Program Committee

Samira Abnar (University of Amsterdam) Lezhi Li (Uber)
David Alvarez Melis (MIT) Anqi Liu (Caltech)
Forough Arabshahi (Carnegie Mellon University) Yin Lou (Ant Financial)
Kamyar Azzizzadenesheli (UC Irvine) David Madras (University of Toronto / Vector Institute)
Gagan Bansal (University of Washington) Sara Magliacane (IBM Research)
Osbert Bastani (University of Pennsylvania) Momin Malik (Berkman Klein Center)
Joost Bastings (University of Amsterdam) Matthew Mcdermott (MIT)
Andrew Beam (Harvard University) Smitha Milli (UC Berkeley)
Kush Bhatia (UC Berkeley) Shira Mitchell ()
Umang Bhatt (Carnegie Mellon University) Tristan Naumann (Microsoft Research)
Cristian Canton (Facebook) Besmira Nushi (Microsoft Research)
Arthur Choi (UCLA) Saswat Padhi (UCLA)
Grzegorz Chrupala (Tilburg University) Emma Pierson (Stanford University)
Sam Corbett-Davies (Facebook) Forough Poursabzi-Sangdeh (Microsoft Research)
Amit Dhurandhar (IBM Research) Manish Raghavan (Cornell University)
Samuel Finlayson (Harvard Medical School, MIT) Ramya Ramakrishnan (MIT)
Tian Gao (IBM Research) Alexander Ratner (Stanford University)
Efstathios Gennatas (UCSF) Andrew Ross (Harvard University)
Siongthye Goh (Singapore Management University) Shibani Santurkar (MIT)
Albert Gordo (Facebook) Prasanna Sattigeri (IBM Research)
Ben Green (Harvard University) Peter Schulam (Johns Hopkins University)
Jayesh Gupta (Stanford University) Ravi Shroff (NYU)
Satoshi Hara (Osaka University) Camelia Simoiu (Stanford University)
Tatsunori Hashimoto (MIT) Sameer Singh (UC Irvine)
He He (NYU) Alison Smith-Renner (University of Maryland)
Fred Hohman (Georgia Institute of Technology) Jina Suh (Microsoft Research)
Lily Hu (Harvard University) Adith Swaminathan (Microsoft Research)
Xiaowei Huang (University of Liverpool) Michael Tsang (University of Southern California)
Yannet Interian (University of San Francisco) Dimitris Tsipras (MIT)
Saumya Jetley (University of Oxford) Berk Ustun (Harvard University)
Shalmali Joshi (Vector Institute) Gilmer Valdes (UCSF)
Yannis Kalantidis (Facebook) Paroma Varma (Stanford University)
Ece Kamar (Microsoft Research) Kush Varshney (IBM Research)
Madian Khabsa (Facebook) Fulton Wang (Sandia National Labs)
Heidy Khlaaf (Adelard) Yang Wang (Uber)
Pang Wei Koh (Stanford University) Fanny Yang (ETH Zurich)
Josua Krause (Accern) Jason Yosinski (Uber)
Ram Kumar (Microsoft / Berkman Klein Center) Muhammad Bilal Zafar (Bosch Center for Artificial Intelligence)
Isaac Lage (Harvard University) Xuezhou Zhang (University of Wisconsin-Madison)
Finnian Lattimore (Australian National University) Xin Zhang (MIT)
Marco Tulio Ribeiro (Microsoft Research)