MATH 076: Mathematics of AI and Large Language Models

MATH 076 (01)

[Topics in Applied Mathematics]

Mathematics of AI and Large Language Models

This course is a mathematical introduction to artificial intelligence and large language models. We will study vectors, matrices, tensors, embeddings, gradient descent, automatic differentiation, attention mechanisms, and generative models. The course will emphasize hands-on mathematical examples, especially simple low-dimensional versions of ideas that appear in modern AI systems.

Course Information

Term: Summer 2026 (June 25, 2026 - August 26, 2026)

Class meetings: Monday, Wednesday, Friday, 2:10–3:15 PM

X-hour: Thursday, 1:20–2:10 PM

Office hours: M 10-11 AM, W 3:30-4:30 PM -- Kemeny 316

Instructor: Mohammad Javad Latifi Jebelli (mohammad.javad.latifi.jebelli@dartmouth.edu)

Grader: Arses Prasai (Arses.Prasai.28@dartmouth.edu)

Interactive Visualizations

The row and column pictures

Visual backpropagation

Hands-on ML with pytorch

Homeworks

Homework 1 - due Friday Jul 10, 7 PM

Homework 2 - due Friday Jul 17, 7 PM

Homework 3 - due Friday Jul 24, 7 PM

Lecture Notes:

Course Grade

Component	Points
Homework	10
Class Activities	10
Quiz	20
Final Project	20
Midterm	30
Final Exam	10
Total	100

Submission of assignments (gradescope)

(1) The class activities are usually due at 7 PM, Friday. If we have multiple activities in the same week, all of them are due on the same Friday.

(2) Homework assignments are usually due at 7 PM, Thursday (if we have one assigned in a given week)

(3) The quiz need to submitted to gradescope right after the completion. The quiz will take place on most Thursdays, 1:40 - 2:10 PM (during X-hour).

Notes:

- Homework and class activities are graded for completion (you don't need to get a complete correct answer to get the points), with no feedback.

- Quiz problems are selected from homework problems of the past week, with possibly a minor modification.

Weekly Plan

Week 1: Overview and Foundations (June 26 – July 3)

Overview of AI, neural networks. Vectors and dot products. Row and column pictures of matrix multiplications. Postulates of a machine that can learn via adjustable parameters. Linear functions as simplest parameterized family of functions. Single layer feed forward neural networks (FFNN). The row picture of FFNN.

Week 2: From Linear Algebra to Linear Layers (July 3 – July 10)

Many pictures of FFNN: pipe picture, row picture, column picture and mixed pictures. High-dimensional geometry. SVD, Low Rank Approximation, Linar Regression. Encoder-decoder models. Linear Autoencoders and connection to SVD.

Week 3: From Dual Numbers to Automatic Differentiation (July 13 – July 17)

Dual numbers. Extension of a real valued function to a dual number valued function. Dual number approach to differentiation and chain rule. Computation graphs, automatic differentiation, and backpropagation.

Week 4: From Multivariate Calculus to Gradient Descent (July 20 – July 24)

Gradients, chain rule, loss functions, gradient descent, and training. Implementing a neural network in PyTorch.

Week 5: From Tensor Product Spaces to Tensor Computation (July 27 – July 31)

Tensors, tensor product spaces, examples of 3-tensors, slices of a tensor, decomposable tensors, dimension and shape of a tensor, reshaping a tensor.

Midterm: The midterm will take place in class on Friday July 31.

Week 6: From Dynamics to LLMs I (August 3 – August 7)

Vector fields on a high-dimensional sphere. Vector fields represented by kernels.

Week 7: From Dynamics to LLMs II (August 10 – August 14)

Many-particle dynamics on the sphere, next-token prediction, and the attention mechanism.

Week 8: Project Presentations (August 17 – August 21)

Project presentations.

Week 9: From Probability to Generative Models (August 24 – August 26)

Time permitting: probability distributions, conditional probability, maximum likelihood, diffusion models, and final project discussions.

Project Ideas

Visualizing Transformer Dynamics*
Explore methods for visualizing how information flows through a transformer during inference or training. Possible projects include attention evolution, token representations, feature trajectories, or layer-wise dynamics to better understand transformer behavior.
Training with a Multi-Component Loss Function (Loss Vector)
Investigate neural network training using a vector-valued loss rather than a single scalar objective. Study optimization strategies, trade-offs between competing objectives, and the geometric interpretation of multi-component loss landscapes.
Naturally Evolving Layerless Neural Networks*
Explore neural network architectures that do not rely on a fixed layer design but instead evolve according to a certain dynamic. Projects may investigate theoretical foundations, training methods, or comparisons with conventional deep networks.
Function Approximation with Residual Neural Networks (e.g., ResNet)*
Study residual neural networks from the perspective of function approximation. Possible directions include approximation theory, convergence properties, expressivity, or the connection between residual networks and differential equations.
The Geometry of Neural Network Function Manifolds*
Investigate the geometric properties of the finite-dimensional manifold of functions represented by a neural network, including curvature and related differential geometric concepts. Projects may explore how this function manifold interacts with training data, optimization, and the loss landscape.
Training Neural Networks to Implement Attention Digraphs
Design and train neural networks that learn directed graph-based attention. The project may examine graph representations.

* Indicates a broad research theme from which multiple individual projects can be developed.

Course Goals

By the end of the course, students should be able to:

Explain the basic mathematical objects used in neural networks and large language models.
Work with vectors, matrices, tensors, gradients, and simple computational graphs.
Connect geometric ideas such as distance, projection, and high-dimensional structure to embeddings and attention.
Describe the role of optimization and automatic differentiation in training AI models.
Build and present a small mathematical or computational project related to AI or generative models.

Accessibility Needs

Students with disabilities who may need disability-related academic adjustments and services for this course are encouraged to see me privately as early in the term as possible. Students requiring disability-related academic adjustments and services must consult the Student Accessibility Services office (Carson Hall, Suite 125, 646-9900). Once SAS has authorized services, students must show the originally signed SAS Services and Consent Form and/or a letter on SAS letterhead to me. As a first step, if students have questions about whether they qualify to receive academic adjustments and services, they should contact the SAS office. All inquiries and discussions will remain confidential.