ECE1513H: Introduction to Machine Learning

Winter 2020 - LEC0101

Overview

An introduction to the basic theory, the fundamental algorithms, and the computational toolboxes of machine learning. The focus is on a balanced treatment of the practical and theoretical approaches, along with hands on experience with relevant software packages. Supervised learning methods covered in the course will include: the study of linear models for classification and regression and neural networks. Unsupervised learning methods covered in the course will include: principal component analysis, k-means clustering, and Gaussian mixture models. Techniques to control overfitting, including regularization and validation, will be covered.

How to contact us?

Do not email any of us individually, but rather send an email to [email protected]. This allows us to redirect your question to the right person. The only exception to this rule is when you would like to discuss something confidential, in which case you can contact Prof. Papernot directly.

  • Instructor: Nicolas Papernot
  • Teaching assistants:

    Schedule and material

    Below is the calendar for this semester course. This is the preliminary schedule, which will be altered as the semester progresses. I will attempt to announce any change to the class, but this webpage should be viewed as authoritative.

    # Date Title Topics Reading Slides Assignment
    1 Jan 6 Introduction Supervised, unsupervised, and reinforcement learning - Slides
    2 Jan 13 Linear regression Vectorization of the optimization problem, gradient descent, overfitting, underfitting, validation Lecture notes
    3 Jan 20 Linear classification Binary classification, perceptron, limits of binary linear classification, Logistic regression, hinge loss. Lecture notes Assignment 1 due
    T Jan 27 JAX Tutorial JAX Solution
    4 Feb 3 Support Vector Machines Multiclass classification. Gradient checking. Support Vector Machines Assignment 2 due
    5 Feb 10 Multilayer Perceptrons Activation functions, feature learning by function composition, expressive power
    - Feb 17 Reading week Assignment 3 due
    6 Feb 24 Gradient descent and Optimization Chain rule, backpropagation algorithm, local optima, saddle points, plateaux, ravines, stochastic gradient descent, momentum
    7 Mar 2 CNNs Convolution layers, pooling layers, associated backprop rules for CNNs, CNN parameter size. Assignment 4 due
    8 Mar 9 Regularization Bias/variance decomposition, data augmentation, limiting capacity, early stopping, dropout, weight decay, ensembles, dropout, hyperparameter tuning.
    9 Mar 16 [3-4pm] Test Lectures 1-8
    [4-5pm] Recurrent Neural Networks and Learning Long-Term Dependencies Recurrent architectures, backpropagation through time, language modeling, neural machine translation, gradients explosion and vanishing, gradient clipping, LSTM.
    10 Mar 23 ResNets and Attention Deep Residual Networks. Attention-based models for machine translation. Assignment 5 due
    11 Mar 30 Optimizing the Input CNN visualizations, adversarial examples, Deep Dream.
    12 Apr 6 Unsupervised Learning Principal component analysis, k-means clustering, and Gaussian mixture models Assignment 6 due
    - Apr 14-28 Exam period

    Assignments

    Some assignments will require you to code using Python, NumPy, and JAX. To alleviate the need for you to purchase and setup a machine with a GPU to accelerate computations, all assignments involving code will be designed to be compatible with the free Colab platform. This platform gives you access to a Jupyter notebook along with a free GPU and does not require any setup.

    Here are some recommended background readings on Python and NumPy.

    Grading

    Grading scheme: Assignments: 40%, Test: 20%, Final: 40%. Your lowest grade to the assignments will be dropped.

    Lateness policy: Assignments are due at the beginning of class at the date posted on the schedule above. All assignments will be assessed a 15% per-day late penalty, up to a maximum of 4 days. Students with legitimate reasons who contact the professor before the deadline may apply for an extension.

    Integrity: Any instance of sharing or plagiarism, copying, cheating, or other disallowed behavior will constitute a breach of ethics. Students are responsible for reporting any violation of these rules by other students, and failure to constitutes an ethical violation that carries with it similar penalties.

    Inclusivity Statement

    All students and faculty at the University of Toronto have a right to learn, work and create in a welcoming, respectful, inclusive and safe environment. In this class we are all responsible for our language, action and interactions. Discriminatory comments or actions of any kind will not be permitted. This includes but is not limited to acts of racism, sexism, Islamophobia, anti-Semitism, homophobia, transphobia, and ableism. As a class we will work together to create an inclusive learning environment and support each other’s learning. If you experience or witness any form of discrimination, please reach out to the Engineering Equity Diversity & Inclusion Action Group online, an academic advisor, a U of T Equity Office, or any U of T Engineering faculty or staff member that you feel comfortable approaching.

    Accommodations

    If you have a learning need requiring an accommodation the University of Toronto recommends that students immediately register at Accessibility Services at www.studentlife.utoronto.ca/as. Location: 4th floor of 455 Spadina Avenue, Suite 400 Voice: 416-978-8060 Fax: 416-978-5729 Email: [email protected] The University of Toronto supports accommodations of students with special learning needs, which may be associated with learning disabilities, mobility impairments, functional/fine motor disabilities, acquired brain injuries, blindness and low vision, chronic health conditions, addictions, deafness and hearing loss, psychiatric disabilities, communication disorders and/or temporary disabilities, such as fractures and severe sprains, recovery from an operation, serious infections or pregnancy complications.

    Mental Health

    As a university student, you may experience a range of health and/or mental health issues that may result in significant barriers to achieving your personal and academic goals. The University of Toronto offers a wide range of free and confidential services and programs that may be able to assist you. We encourage you to seek out these resources early and often. Health & Wellness Resources: undergrad.engineering.utoronto.ca/advising-and-wellness/health-wellness/ U of T Health & Wellness Website: studentlife.utoronto.ca/hwc If, at some point during the year, you find yourself feeling distressed and in need of more immediate support, visit the Feeling Distressed Webpage: www.studentlife.utoronto.ca/feeling-distressed, for more campus resources. Off campus, immediate help is available 24/7 through Good2Talk, a post-secondary student helpline at 1-866-925-5454. All students in the Faculty of Engineering have an Academic Advisor who can advise on academic and personal matters. You can find your department’s Academic Advisor here: uoft.me/engadvising

    Acknowledgments

    Material used in this course is adapted from several prior iterations as well as similar courses taught by others. This includes CSC321 by Prof. Grosse.