Nicolas Papernot
I am an Assistant Professor in the Department of Electrical and Computer Engineering and the Department of Computer Science at the University of Toronto. I am also a faculty member at the Vector Institute where I hold a Canada CIFAR AI Chair, and a faculty affiliate at the Schwartz Reisman Institute.
My research interests are at the intersection of security, privacy, and machine learning. If you would like to learn more about my research, I recommend reading the blog posts I co-authored on cleverhans.io, for example about proof-of-learning, collaborative learning beyond federation, dataset inference, machine unlearning, differentially private ML, or adversarial examples.
My research has been cited in the press, including the New York Times, Popular Science, and Wired. I earned my Ph.D. in Computer Science and Engineering at the Pennsylvania State University, working with Prof. Patrick McDaniel and supported by a Google PhD Fellowship. Upon graduating, I spent a year at Google Brain where I still spend some of my time.
Email: [email protected]
Office: Pratt 484E and MaRS Suite 710
Mail/Packages: 10 King's College Road, Room SFB540, Toronto, ON M5S 3G4, Canada
Selected publications
A complete list of publications is available in my CV.
- On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning. Anvith Thudi, Hengrui Jia, Ilia Shumailov, Nicolas Papernot. preprint
- Hyperparameter Tuning with Renyi Differential Privacy. Nicolas Papernot, Thomas Steinke. preprint
- Bad Character Injection: Imperceptible Attacks on NLP Models. Nicholas Boucher, Ilia Shumailov, Ross Anderson, Nicolas Papernot. Proceedings of the 43rd IEEE Symposium on Security and Privacy, San Francisco, CA (conditional acceptance). conference
- SoK: Machine Learning Governance. Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, Nicolas Papernot. preprint
- Towards More Robust Keyword Spotting for Voice Assistants. Shimaa Ahmed, Ilia Shumailov, Nicolas Papernot, Kassem Fawaz. Proceedings of the 31st USENIX Security Symposium. conference
- Manipulating SGD with Data Ordering Attacks. Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A. Erdogdu, Ross Anderson. Proceedings of the 35th Conference on Neural Information Processing Systems. conference
- Data-Free Model Extraction. Jean-Baptiste Truong, Pratyush Maini, Robert Walls, Nicolas Papernot. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN. conference
- Proof-of-Learning: Definitions and Practice. Hengrui Jia, Mohammad Yaghini, Christopher A. Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, Nicolas Papernot. Proceedings of the 42nd IEEE Symposium on Security and Privacy, San Francisco, CA. conference
- Entangled Watermarks as a Defense against Model Extraction. Hengrui Jia, Christopher A. Choquette-Choo, Varun Chandrasekaran, Nicolas Papernot. Proceedings of the 30th USENIX Security Symposium. conference
- Sponge Examples: Energy-Latency Attacks on Neural Networks. Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, Ross Anderson. Proceedings of the 6th IEEE European Symposium on Security and Privacy, Vienna, Austria. conference
- CaPC Learning: Confidential and Private Collaborative Learning. Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao Wang. Proceedings of the 9th International Conference on Learning Representations. conference
- Dataset Inference: Ownership Resolution in Machine Learning. Pratyush Maini, Mohammad Yaghini, Nicolas Papernot. Proceedings of the 9th International Conference on Learning Representations. conference (+spotlight)
- Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings. Vinith Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. conference
- Tempered Sigmoids for Deep Learning with Differential Privacy. Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, Ulfar Erlingsson. Proceedings of the 35th AAAI Conference on Artificial Intelligence. conference
- Machine Unlearning. Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot. Proceedings of the 42nd IEEE Symposium on Security and Privacy, San Francisco, CA. conference
- Thieves of Sesame Street: Model Extraction on BERT-based APIs. Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, Mohit Iyyer. Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia. conference
- Analyzing and Improving Representations with the Soft Nearest Neighbor Loss. Nicholas Frosst, Nicolas Papernot, Geoffrey Hinton. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA. conference
- Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning. Nicolas Papernot and Patrick McDaniel. technical report
- Scalable Private Learning with PATE. Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, Ulfar Erlingsson. Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada. conference
- Towards the Science of Security and Privacy in Machine Learning. Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael Wellman. Proceedings of the 3rd IEEE European Symposium on Security and Privacy, London, UK. conference
- Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. Nicolas Papernot, Martin Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Proceedings of the 5th International Conference on Learning Representations, Toulon, France. conference (+best paper)
- Practical Black-Box Attacks against Machine Learning. Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z.Berkay Celik, and Ananthram Swami. Proceedings of the 2017 ACM Asia Conference on Computer and Communications Security, Abu Dhabi, UAE. conference
- Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples. Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. technical report
- The Limitations of Deep Learning in Adversarial Settings. Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. Proceedings of the 1st IEEE European Symposium on Security and Privacy, Saarbrucken, Germany. conference
Research group
Current students and postdocs
- Roei Schuster: Postdoctoral Fellow (starting Fall 2021)
- Ilia Shumailov: Postdoctoral Fellow (starting Fall 2021, co-advised with Kassem Fawaz)
- Aditi Misra: Engineering Science student (started Fall 2021)
- Franziska Boenisch: Research Intern (Summer 2021 - Fall 2021)
- Hongyu (Charlie) Chen: Engineering Science student (Fall 2021 - Summer 2022)
- Sierra Wyllie: Engineering Science student (started Summer 2021)
- Muhammad Ahmad Kaleem: Engineering Science student (started Summer 2021)
- Armin Ale: Engineering Science student (Summer 2021 - Fall 2022)
- Emmy Fang: MS student (started Fall 2021)
- Ali Shahin Shamsabadi: Research Intern (Winter 2021 - Fall 2021)
- Anvith Thudi: Mathematics Specialist Undergraduate student (started Fall 2020)
- Adam Dziedzic: Postdoctoral Fellow (started Fall 2020)
- Mohammad Yaghini: PhD student (started Fall 2020)
- Natalie Dullerud: MS student (started Fall 2020)
- Stephan Rabanser: PhD student (started Fall 2020)
- Jonas Guan: PhD student (started Fall 2020)
- Jiaqi Wang: MASc student (started Fall 2020, co-advised with David Lie)
- Nick Jia: MASc student (started Fall 2020)
- Mingyue Yang: PhD student (started Winter 2020, co-advised with David Lie)
Past students and postdocs
- Aisha Alaagib: Research Intern (Summer 2021) currently PhD student at MILA
- Steven Xia: Undergraduate student (Fall 2020 - Summer 2021, co-advised with Shurui Zhou) currently PhD student at UIUC
- Jin Zhou: Engineering Science student (Fall 2020 - Summer 2021) currently PhD student at Cornell
- Lucy Lu: Engineering Science student (Fall 2020 - Summer 2021) currently MS student at Stanford
- Marko Huang: Engineering Science student (Fall 2020 - Summer 2021) currently MS student at University of Toronto
- Gabriel Deza: Engineering Science student (Fall 2020 - Summer 2021) currently PhD student at UC Berkeley
- Tejumade Afonja: Research Intern (Summer 2020) currently MS student at Saarland University
- Milad Nasr: Google Brain Intern (Summer 2020, co-hosted with Nicholas Carlini) currently PhD student at UMass
- Lorna Licollari: Research Intern (Summer 2020) currently Engineering Science student at University of Toronto
- Pratyush Maini: Research Intern (Summer 2020) currently PhD student at CMU
- Yunxiang Zhang: Research Intern (Spring 2020) currently PhD student at Chinese University of Hong Kong
- Saina Asani: Research Assistant (Winter 2020 - Summer 2020) currently AI Researcher at Huawei
- Laura Zhukas: Undergraduate Student Researcher (Fall 2019) currently BASc student at the University of Waterloo
- Christopher Choquette-Choo: Engineering Science student (Fall 2019 - Summer 2020) currently Google AI Resident
- Baiwu Zhang: MEng student (Fall 2019 - Summer 2020) currently ML Engineer at Twitter
- Varun Chandrasekaran: Visiting PhD student (Fall 2019) currently PhD student at University of Wisconsin
- Vinith Suriyakumar: MS student (Fall 2019 - Summer 2021, co-advised with M. Ghassemi and A. Goldenberg) currently PhD student at MIT
- Lucas Bourtoule: MASc student (started Fall 2019) currently Data Scientist at Total Energies
- Adelin Travers: PhD student (Fall 2019 - Summer 2021, co-advised with David Lie) currently Senior Pentester at Verizon
- Hadi Abdullah: Google Intern (Summer 2019, co-hosted with Damien Octeau) currently PhD student at University of Florida
- Matthew Jagielski: Google Brain intern (Summer 2019) currently PhD student at Northeastern University
Information for prospective graduate students and postdocs
- If you are interested in joining my research group as a graduate student, apply to the CS or ECE (select "software systems" field) program. Unfortunately, I cannot respond to all prospective graduate students, so the best time is to contact me after you submitted your application.
- If you are interested in joining my research group as a postdoc, please send me an email directly with your CV and research statement.
Research Talks
Upcoming
Here is a list of talks I will be giving. Feel free to reach out if you will be attending one of these events and would like to meet.
- 11/2021 - CAMLIS 2021 keynote
Past Recorded Talks
These video resources are a good overview of my research interests.
Tempered Sigmoids for DP-SGD
Trustworthy ML
Deepfakes
Lecture on ML security and privacy
Privacy-preserving ML
Adversarial examples
Blog Posts
Here is a list of blog posts discussing some of the research questions I'm interested in:
- How to deploy machine learning with differential privacy? (DifferentialPrivacy.org)
- Arbitrating the integrity of stochastic gradient descent with proof-of-learning
- Beyond federation: collaborating in ML with confidentiality and privacy
- Is this model mine?
- Why we should regulate information about persons, not personal information
- To guarantee privacy, focus on the algorithms, not the data
- Teaching Machines to Unlearn
- In Model Extraction, Don’t Just Ask How?: Ask Why?
- How to steal modern NLP systems with gibberish?
- The academic job search for computer scientists in 10 questions
- How to know when machine learning does not know
- Machine Learning with Differential Privacy in TensorFlow
- Privacy and machine learning: two unexpected allies?
- The challenge of verification and testing of machine learning
- Is attacking machine learning easier than defending it?
- Breaking things is easy
Teaching
- [Fall 2021] ECE1784H/CSC2559H: Trustworthy Machine Learning
- [Fall 2021] ECE421H: Introduction to Machine Learning (see Quercus for course details)
- [Fall 2020] ECE421H: Introduction to Machine Learning (see Quercus for course details)
- [Fall 2020] ECE1513H: Introduction to Machine Learning (see Quercus for course details)
- [Winter 2020] ECE1513H: Introduction to Machine Learning (see Quercus for course details)
- [Fall 2019] ECE1784H: Trustworthy Machine Learning