ECE1784H: Trustworthy Machine Learning (Fall 2019)

Overview

• Instructor: Nicolas Papernot ([email protected])
• Location: AB107
• Time: Mondays 1-3pm
• Office hours: Wednesdays 1:30-3:30pm
• Office location: Pratt 484E

The deployment of machine learning in real-world systems calls for a set of complementary technologies that will ensure that machine learning is trustworthy. Here, the notion of trust is used in its broad meaning: the course covers different topics in emerging research areas related to the broader study of security and privacy in machine learning. Students will learn about attacks against computer systems leveraging machine learning, as well as defense techniques to mitigate such attacks.

The course assumes students already have a basic understanding of machine learning. Students will familiarize themselves with the emerging body of literature from different research communities investigating these questions. The class is designed to help students explore new research directions and applications. Most of the course readings will come from both seminal and recent papers in the field. No textbooks are required for this class. Notes and slides, as well as research papers, will make up the material used in this course. Links to these will be provided in the schedule below.

Schedule and material

Below is the calendar for this semester course. This is the preliminary schedule, which will be altered as the semester progresses. I will attempt to announce any change to the class, but this webpage should be viewed as authoritative. If you have any questions, please contact me.

#	Date	Topic	Slides/Notes	Reading / Assignment
1	Sep 09	Overview & motivation	slides	Reading: Saltzer and Schroeder, The Protection of Information in Computer Systems.
-	Sep 13	Exam		-
2	Sep 16	Training-time integrity (attacks & defenses)		Reading: Nelson et al., Exploiting Machine Learning to Subvert Your Spam Filter. Rubinstein et al., ANTIDOTE: Understanding and Defending against Poisoning of Anomaly Detectors. Koh and Liang, Understanding Black-box Predictions via Influence Functions. Jagielski et al., Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. Shafahi et al., Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks. Wang et al., Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks.
3	Sep 23	Test-time integrity (attacks)		Mandatory Reading: Szegedy et al., Intriguing properties of neural networks. Papernot et al., Practical Black-Box Attacks against Machine Learning. Xu et al., Automatically Evading Classifiers. Hong et al., Terminal Brain Damage: Exposing the Graceless Degradation in Deep Neural Networks Under Hardware Fault Attacks. Optional Reading: Lowd and Meek, Good Word Attacks on Statistical Spam Filters. Biggio et al., Evasion Attacks against Machine Learning at Test Time
4	Sep 30	Test-time integrity (defenses)		Reading: Dalvi et al., Adversarial Classification. Tramer et al., Ensemble Adversarial Training: Attacks and Defenses. Wong and Kolter, Provable defenses against adversarial examples via the convex outer adversarial polytope. Lecuyer et al., Certified Robustness to Adversarial Examples with Differential Privacy. Optional Reading: Geirhos et al., ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.
5	Oct 07	Confidentiality (of the model)		Reading: Lowd and Meek, Adversarial Learning. Tramer et al., Stealing Machine Learning Models via Prediction APIs. Batina et al., CSI NN: Reverse Engineering of Neural Network Architectures Through Electromagnetic Side Channel. Jagielski et al., High-Fidelity Extraction of Neural Network Models. Optional Reading: Chandrasekaran et al., Model Extraction and Active Learning
	Oct 14	Thanksgiving
6	Oct 21	Privacy attacks		Reading: Narayanan and Shmatikov, Robust De-anonymization of Large Sparse Datasets. Shokri et al., Membership Inference Attacks against Machine Learning Models. Carlini et al., The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. Song and Shmatikov, Overlearning Reveals Sensitive Attributes.
7	Oct 28	Differential privacy	slides	Reading: Dwork et al., Calibrating Noise to Sensitivity in Private Dat Analysis. Abadi et al., Deep Learning with Differential Privacy. Papernot et al., Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data. Ilya Mironov - Renyi Differential Privacy (recording of talk at DIMACS).
	Nov 04	Projects
8	Nov 11	Confidentiality (of the data)		Reading: Ohrimenko et al., Oblivious Multi-Party Machine Learning on Trusted Processors. Juvekar et al., GAZELLE: A Low Latency Framework for Secure Neural Network Inference. Grover et al., PRIVADO: Practical and Secure DNN Inference with Enclaves. Nasr et al., Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks. Optional Reading: McMahan et al., Communication-Efficient Learning of Deep Networks from Decentralized Data.
9	Nov 18	Safety		Reading: Amodei et al., Concrete Problems in AI Safety. Kurakin et al., Adversarial Examples in the Physical World. Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. Pei et al., DeepXplore: Automated Whitebox Testing of Deep Learning Systems. Optional Reading: Tsutomu Matsumoto, Impact of Artificial "Gummy" Fingers on Fingerprint Systems.
10	Nov 25	Fairness & Ethics	slides	Reading: Dwork et al., Fairness Through Awareness. Caliskan et al., Semantics derived automatically from language corpora contain human-like biases. Speicher et al., Potential for Discrimination in Online Targeted Advertising. Kumar et al., Law and Adversarial Machine Learning.
11	Dec 02	Poster session

Assignments

LaTeX template for assignments: you can find an example template for writing the reading summaries and class notes on Overleaf. You can also use that service to write up your assignments if you don't want to install latex on your own machine.

Reading summary: A 1 page summary of reading assigned is due each class (starting from week 2 and onwards). A physical copy should be turned in before the beginning of class. The summary should cover the following: (a) what did the papers do well?, (b) where did the papers fall short?, (c) what did you learn from these papers?, and (d) what questions do you have about the papers?

Paper presentation: starting from week 2, a team of students will present the papers assigned for reading each week. The team may choose an appropriate format (e.g., slides, interactive demos or code tutorials, ...) for this presentation with the only requirements being that the presentation should (a) involve the class in active discussions, (b) cover all papers assigned for reading, and (c) last no more than 1h30mn including discussions.

Class notes: Another team of students will be charged with writing notes synthesizing the content of the presentation and class discussion.

Research projects: Students will work on a course-long research project. Each project will be presented in the form of a poster on Dec 02.

Grading

Grading scheme: 15% exam, 25% paper presentation, 10% paper summaries, 15% class notes, 35% research project.

Class participation: Course lectures will be driven by the contents of assigned papers. However, students are going to be required to (i) turn in a 1 page summary of reading assigned each week, (ii) participate in discussions of the paper content during each class, and (iii) write notes synthesizing the content of class for one class. Hence, the students' ability to exhibit comprehension of papers is essential to a passing grade.

Lateness policy: 1 page summaries of reading assigned each week will not be accepted late (students will be assigned a 0 for that week). All other assignments (i.e., presentation slides, notes synthesizing the content of class, and project reports) will be assessed a 10% per-day late penalty, up to a maximum of 4 days (2 days for presentation slides). Students with legitimate reasons who contact the professor before the deadline may apply for an extension.

Integrity: Any instance of sharing or plagiarism, copying, cheating, or other disallowed behavior will constitute a breach of ethics. Students are responsible for reporting any violation of these rules by other students, and failure to constitutes an ethical violation that carries with it similar penalties.

Ethics statement

This course covers topics in personal and public privacy and security. As part of this investigation we will explore technologies whose abuse may infringe on the rights of others. As an instructor, I rely on the ethical use of these technologies. Unethical use may include circumvention of existing security or privacy measurements for any purpose, or the dissemination, promotion, or exploitation of vulnerabilities of these services. Exceptions to these guidelines may occur in the process of reporting vulnerabilities through public and authoritative channels. Any activity outside the letter or spirit of these guidelines will be reported to the proper authorities and may result in dismissal from the class. When in doubt, please contact the course professor for advice. Do not undertake any action which could be perceived as technology misuse anywhere and/or under any circumstances unless you have received explicit permission from the instructor.