EE212 Mathematical Foundations for Machine Learning and Data Science

Fall 2021

Department of Electrical Engineering
Syed Babar Ali School of Science and Engineering
Lahore University of Management Sciences.

Announcements

• (Sep. 12) Assignment 01 has been posted (due on September 26).

• (Sep. 05) Welcome to EE212. Course outline has been posted.
  
  

Course Overview

Machine Learning and Data Science are being used these days in a variety of applications including, but not limited to, forecasting in economics and finance, predicting anomalies or signal analysis in engineering, identification of speaker in acoustics, detection of cosmic bubbles in astrophysics and diagnosis in medical imaging.

While machine learning and data science have enabled many success stories, and tools are readily available to analyse data or design machine learning systems, the strong mathematical foundations in these areas are of significant importance to understand, review, analyse and evaluate the technical details of the machine learning systems and data science algorithms that are usually abstracted away from the user. This course focuses on the mathematical foundations that are essential to build an intuitive understanding of the concepts related to Machine Learning and Data Science.

Topics covered are

• Linear Algebra: vectors and matrices, vector spaces, system of linear equations, eigen-value decomposition, singular value decomposition, regression, least-squares, regularization

• Calculus: Multivariate calculus and differentials for optimization, gradient descent

• Probability: probability axioms, Bayes rule, random variable, probability distributions

• Statistics: descriptive stats, inferential stats, statistical tests

• Introduction to Neural Networks: single and multi-layer perceptron(s), feedforward and feedback networks

• Application to machine learning and data science: principal component analysis (PCA), time series forecasting, clustering etc

• Hands-on exercises: Implementation of the exercises will be carried out in Python
  

Course Overview Video

Administrative Details

• Course Outline (Click to download)
  
  

• Suggested Books:
  

  • Reference 1 (click to download pdf): S.Boyd and L. Vandenberghe. Introduction to Applied Linear Algebra - Vectors, Matrices, and Least Squares. Cambridge University Press, 2019

  • Reference 2 (click to download pdf): M. P. Deisenroth, A. A. Faisal and Cheng Soon Ong. Mathematics for Machine Learning. Cambridge University Press, 2019

  • Reference 3: G. Strang. Introduction to Linear Algebra. 2016

  • Reference 4: J. A. Gubner, Probability and Random Processes for Electrical and Computer Engineers, Cambridge University Press, 2006.

  • Reference 5: S. L. Miller and D. Childers, Probability and Random Processes: With Applications to Signal Processing and Communications.

  • Reference 6: A. Papoulis and S.U. Pillai, Probability, Random Variables, and Stochastic Processes.
    
    

• Office Hours and Contact Information
  

  • Instructor: Zubair Khalid (zubair.khalid@lums.edu.pk), Office hours: Tuesday, Thursday 5-6 pm
    

  • Lead Teaching Assistant: Zainab Imran (21100153@lums.edu.pk), Office hours: Friday 10:30 am to 12:30 pm
    

  • Teaching Assistant: Sana Jabbar (18060039@lums.edu.pk), Office hours: TBA
    

  • Teaching Assistant: Hamza Khalid (22100118@lums.edu.pk), Office hours: TBA
    

Grading Distribution

• Assignments, 20 %

• Programming Assignments, 10 %

• Quizzes, 15 %

• Mid-Exam + Viva, 25 %

• Final Exam + Viva, 30 %
  

Exams

• Mid-Exam
  
  

• Final-Exam
  
  

Tutorials

• Tutorial 01
  

• Tutorial 02
  

• Tutorial 03
  

• Tutorial 04
  

• Tutorial 05
  

• Tutorial 06
  

Assignments

Programming Assignments

• Programming Assignment 0 (Pre-Lab, must be completed before Lab 01)
  

• Programming Assignment 01

• Programming Assignment 02, Image for Assignment
  

• Programming Assignment 03
  

• Programming Assignment 04 (Data-sets available on LMS)
  
  
  

Lecture Plan

• Weeks 01 and 02 (Lecture Notes)

  • Course Introduction

  • Operations on vectors: linear combination, norm, inner prooduct, angle, distance, correlation coefficient

  • Span, basis, linear independence, orthonormal vectors, vector spaces, Gram-Schmidt orthogonalization
    

• Weeks 03 and 04 (Lecture Notes)

  • Matrices Notation, Application Examples and Basic Operations

  • Matrix-vector product, Interpretations, Application Examples,Matrix-matrix product

  • Systems of Linear Equations, Formulation, Inverses, Left-inverse, Right-inverse, Inverse, Pseudo-inverse, Connection with the linear equations

• Weeks 05 and 06 (Lectures 09-13)

  • EigenValue Decomposition (EVD) (Lecture Notes)

  • Curse of Dimensionality and Principal Component Analysis (Application of EVD) (Lecture Notes)

  • Singular Value Decomposition (SVD) (Lecture Notes)

  • Least-squares Formulation (Lecture Notes)

• Weeks 07 and 08 (Lectures 14-16)

  • Calculus module: Functions, Derivatives, Gradient, Hessian, Jacobian, Anti-Derivatives (Lecture Notes)

• Week 09, 10 and 11 (Lectures 17-21)

  • Probability Theory overview, Probability models, Axioms of probability, Conditional probability Bayes theorem, Law of total probability (Lecture Notes)

  • Independence, Combinatorics

  • Discrete random variable, probability mass function, Continuous random variable, probability density function (Lecture Notes)

  • Introduction to Inference (Lecture Notes)

• Week 12 (Lectures 22-24)

  • Overview of supervised learning, ML nomenclature, problem setup and train-test split (Lecture Notes)

• Week 13 (Lectures 25-26)

  • kNN Algorithm: Overview and Analysis (Lecture Notes)

  • Analysis and Evaluation of Classifier’s Performance (Lecture Notes)

• Week 14 (Lectures 27-28)

  • Overview of Perceptron Classifier, Logistic Regression and Neural Networks (Lecture Notes)