Math for AI & Data Science

Focus · Linear Algebra, Calculus & Probabilistic ML Target Level · Grades 11–12 / University STEM Bridge Core Chapters · 8 Foundational Units Interactive Explorers · 4 Visual ML Explorers

✓ Official Syllabus Verified Last reviewed against official syllabus: 2025–2026 Academic Year Standards & Rigor ↗

Vectors, Norms & High-Dimensional Embeddings

🤖 AI Math Unit 1 — Vectors & Geometric Embeddings Applied STEM · Vector spaces & column vectors · L1 (Manhattan) and L2 (Euclidean) norms · Dot products & vector projections · Cosine similarity in NLP & vector search · Word and token embeddings (Word2Vec / Transformer)

In Artificial Intelligence and Natural Language Processing, text, images, and audio are mapped to high-dimensional numerical vectors called embeddings. We explore vector geometry, norms, and distance metrics used in modern vector databases.

📊 Math for AI & Data Science — Exam Blueprint & Domain Weighting

Vectors & Embeddings

~25% MATHAI

Matrices & Neural Layers

~25% MATHAI

Gradient Descent & Loss

~25% MATHAI

Probability, Bayes & PCA

~25% MATHAI

Official curriculum weighting and examination domain distribution for Math for AI & Data Science.

● Definition 1.1 — Vector Norms & Distances

For a vector

\mathbf{x} = [x_1, x_2, \dots, x_n]^T \in \mathbb{R}^n

:
1. Euclidean ( $L_2$ ) Norm:

\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^n x_i^2} = \sqrt{\mathbf{x}^T\mathbf{x}}

. Measures straight-line geometric distance.
2. Manhattan ( $L_1$ ) Norm:

\|\mathbf{x}\|_1 = \sum_{i=1}^n |x_i|

. Used in Lasso regularization (L1 sparsity).
3. Max ( $L_\infty$ ) Norm:

\|\mathbf{x}\|_\infty = \max_i |x_i|

. Used in adversarial robustness attacks (FGSM).

● Deep Conceptual Intuition — Vectors, Norms & High-Dimensional Embeddings

Linear Structure, Inner Products & Geometric Transformations: In $\mathbb{R}^n$ , the dot product $\mathbf{u}\cdot\mathbf{v} = \|\mathbf{u}\|\|\mathbf{v}\|\cos\theta = \sum_{i=1}^n u_i v_i$ encodes orthogonality ( $\mathbf{u}\perp\mathbf{v} \iff \mathbf{u}\cdot\mathbf{v} = 0$ ) and orthogonal projection $\operatorname{proj}_{\mathbf{v}}\mathbf{u} = \dfrac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{v}\|^2}\mathbf{v}$ . For a matrix transformation $A \in \mathbb{R}^{n\times n}$ , $|\det(A)|$ measures the local volume scaling factor, and $A$ is invertible if and only if $\det(A) \ne 0$ (equivalently, all eigenvalues $\lambda_i \ne 0$ ).

Coordinate grid showing geometric action of a linear transformation mapping △ABC to its image △A'B'C'.

⊙ Explorer · Embedding Space: Cosine Similarity

\cos \theta

L_{1}

vs.

L_{2}

Norm Balls

interactive

Live numerical table for Embedding Space: Cosine Similarity $\cos \theta$ & $L_{1}$ vs. $L_{2}$ Norm Balls
Parameter / Metric	Value

Embedding

u

Angle $\theta_u$ (°) = 30

Embedding

v

Angle $\theta_v$ (°) = 75

Cosine Similarity

\cos(\Delta \theta)

\cos \Delta \theta = 0.707

Euclidean Distance

\|u - v\|

₂

\|u-v\|_{2} = 0.765

Manhattan Distance

\|u - v\|

₁

\|u-v\|_{1} = 1.073

Compare two embedding vectors $u$ and $v$ on the unit $L_{2}$ circle (indigo) and unit $L_{1}$ diamond (dashed gray), tracking their dot-product cosine similarity $\cos(\theta_u - \theta_v)$ .

Vectors, Norms & High-Dimensional Embeddings — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Vectors, Norms & High-Dimensional Embeddings.

Dot Product

\mathbf{u} \cdot \mathbf{v} = \sum_{i=1}^n u_i v_i = \mathbf{u}^T\mathbf{v}

Cosine Similarity

\cos\theta = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|} \in [-1, 1]

Cosine Distance

D_{\cos}(\mathbf{u}, \mathbf{v}) = 1 - \cos\theta

Orthogonality

\mathbf{u} \perp \mathbf{v} \iff \mathbf{u}^T\mathbf{v} = 0 \implies \cos\theta = 0

◆ Why It Works — First-Principles Derivation (Vectors, Norms & High-Dimensional Embeddings)

Why Orthogonal Projection & Eigenvalue Characteristic Equations Work: Decompose $\mathbf{u} = c\mathbf{v} + \mathbf{w}$ where $\mathbf{w} \perp \mathbf{v}$ . Taking the dot product with $\mathbf{v}$ uses $\mathbf{w}\cdot\mathbf{v} = 0$ to isolate $c$ :

\mathbf{u}\cdot\mathbf{v} = c(\mathbf{v}\cdot\mathbf{v}) + 0 \implies c = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{v}\|^2}

For eigenvalues, $A\mathbf{v} = \lambda\mathbf{v}$ with $\mathbf{v} \ne \mathbf{0}$ is equivalent to $(A - \lambda I)\mathbf{v} = \mathbf{0}$ . A homogeneous linear system has a non-trivial nullspace vector $\mathbf{v} \ne \mathbf{0}$ if and only if the matrix $A - \lambda I$ is singular, yielding the characteristic polynomial $\det(A - \lambda I) = 0$ .

Worked Example 1.A — Cosine Similarity Between Word Embeddings

In an NLP embedding model, the word vectors for 'king' and 'queen' in a 3-dimensional subspace are $\mathbf{u} = [0.8, 0.6, 0.0]^T$ and $\mathbf{v} = [0.6, 0.8, 0.0]^T$ . An unrelated word 'apple' has vector $\mathbf{w} = [0.0, 0.1, 0.99]^T$ . Calculate the cosine similarity between 'king' and 'queen', and between 'king' and 'apple'.

Solution:
1. Norm of $\mathbf{u}$ : $\|\mathbf{u}\| = \sqrt{0.8^2 + 0.6^2 + 0^2} = \sqrt{0.64 + 0.36} = 1.0$ .
2. Norm of $\mathbf{v}$ : $\|\mathbf{v}\| = \sqrt{0.6^2 + 0.8^2 + 0^2} = 1.0$ .
3. Dot product: $\mathbf{u} \cdot \mathbf{v} = (0.8)(0.6) + (0.6)(0.8) + (0)(0) = 0.48 + 0.48 = 0.96$ .
Cosine similarity: $\cos\theta_{uv} = \frac{0.96}{(1.0)(1.0)} = 0.96$ (very high semantic similarity!).

4. Now check 'king' and 'apple':
$\mathbf{u} \cdot \mathbf{w} = (0.8)(0) + (0.6)(0.1) + (0)(0.99) = 0.06$ .
Norm $\|\mathbf{w}\| = \sqrt{0^2 + 0.01 + 0.9801} = \sqrt{0.9901} \approx 0.995$ .
Cosine similarity: $\cos\theta_{uw} = \frac{0.06}{(1.0)(0.995)} \approx 0.0603$ (near orthogonal, no semantic relation).
Answer: $\cos\theta_{uv} = 0.96$ ; $\cos\theta_{uw} \approx 0.060$ .

Worked Example

B

— Vectors, Norms & High-Dimensional Embeddings (Easy Application)

Problem (Easy): Consider a simple logistic regression model where the decision boundary is defined by the equation $3x + 4y - 12 = 0$ . This equation represents a linear boundary in a 2D feature space. What is the y-intercept of this decision boundary?

Step-by-Step Solution: The equation of the decision boundary is given by $3x + 4y - 12 = 0$ . To find the y-intercept, we set $x = 0$ and solve for $y$ . \nSubstituting $x = 0$ into the equation: \n $3(0) + 4y - 12 = 0$ $4y - 12 = 0$ $4y = 12$ $y = \frac{12}{4}$ $y = 3$ Therefore, the y-intercept of the decision boundary is 3.

Worked Example

C

— Vectors, Norms & High-Dimensional Embeddings (Medium Application)

Problem (Medium): A linear transformation is defined by the matrix $M = \begin{pmatrix} 2 & 1 \\ 3 & 4 \end{pmatrix}$ . If an input vector is $v = \begin{pmatrix} x \\ 1 \end{pmatrix}$ , and the output vector after the transformation is $w = \begin{pmatrix} 5 \\ 7 \end{pmatrix}$ , what is the value of $x$ ?

Step-by-Step Solution: The linear transformation is given by $w = Mv$ . Substituting the given matrices and vectors, we have: $\begin{pmatrix} 3 \\ 7 \end{pmatrix} = \begin{pmatrix} 2 & 1 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} x \\ 1 \end{pmatrix}$ Performing the matrix multiplication on the right side gives: $\begin{pmatrix} 3 \\ 7 \end{pmatrix} = \begin{pmatrix} 2x + 1(1) \\ 3x + 4(1) \end{pmatrix} = \begin{pmatrix} 2x + 1 \\ 3x + 4 \end{pmatrix}$ Equating the corresponding components of the vectors, we get two equations: 1. $3 = 2x + 1$ 2. $7 = 3x + 4$ Solving the first equation for $x$ : $3 - 1 = 2x$ $2 = 2x$ $x = \frac{2}{2}$ $x = 1$ Solving the second equation for $x$ : $7 - 4 = 3x$ $3 = 3x$ $x = \frac{3}{3}$ $x = 1$ Both equations yield $x=1$ . Therefore, the value of $x$ is 1.

Worked Example

D

— Vectors, Norms & High-Dimensional Embeddings (Hard Application)

Problem (Hard): Consider two vectors, $\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}$ . What is the vector representing $2\mathbf{u} - \mathbf{v}$ ?

Step-by-Step Solution: To find the vector $2\mathbf{u} - \mathbf{v}$ , we first compute $2\mathbf{u}$ and then subtract $\mathbf{v}$ from the result. $2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}$ . Now, we subtract $\mathbf{v}$ from $2\mathbf{u}$ : $2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}$ . Therefore, the resulting vector is $\begin{bmatrix} 5 \\ 10 \end{bmatrix}$ .

⚠️ Common Exam Traps — Vectors, Norms & High-Dimensional Embeddings

1. Non-Commutativity of Matrix Multiplication & Cross Products: On Mathematics for Artificial Intelligence & Machine Learning, assuming $AB = BA$ (so expanding $(A+B)^2$ as $A^2 + 2AB + B^2$ instead of $A^2 + AB + BA + B^2$ ) or forgetting $\mathbf{u}\times\mathbf{v} = -(\mathbf{v}\times\mathbf{u})$ invalidates matrix proofs.

2. Using Position Vectors Instead of Direction Vectors for Angles: To find the angle at vertex $B$ of triangle $ABC$ , you must use tail-to-tail vectors $\overrightarrow{BA}$ and $\overrightarrow{BC}$ , never $\overrightarrow{AB}\cdot\overrightarrow{BC}$ (which finds the exterior supplementary angle $180^\circ - B$ ).

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Vectors, Norms & High-Dimensional Embeddings)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Computing angle at $B$ using $\overrightarrow{AB}\cdot\overrightarrow{BC}$	Vectors are head-to-tail instead of tail-to-tail, giving $180^\circ - \theta$ .	Reverse $\overrightarrow{AB}$ so both emanate from $B$ : use $\overrightarrow{BA}\cdot\overrightarrow{BC}$ .
Using the same parameter $t$ for two intersecting 3D lines $\mathbf{r}_1(t) = \mathbf{r}_2(t)$	Tests for a simultaneous collision at time $t$ rather than geometric intersection of paths.	Assign distinct parameters $\lambda$ and $\mu$ to the two lines before equating components.
Assuming $(AB)^{-1} = A^{-1}B^{-1}$	Multiplying $(AB)(A^{-1}B^{-1})$ does not cancel inner factors when $AB \ne BA$ .	Use the shoe-sock reversal law: $(AB)^{-1} = B^{-1}A^{-1}$ .

▲ Exam Pacing & Verification Strategy — Vectors, Norms & High-Dimensional Embeddings

Fast Matrix & Vector Verification (Mathematics for Artificial Intelligence & Machine Learning): Check every $2\times 2$ or $3\times 3$ eigenvalue computation using the trace and determinant invariants: $\sum \lambda_i = \operatorname{tr}(A)$ (sum of diagonal entries) and $\prod \lambda_i = \det(A)$ .

[Practice 1 · Easy] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

[Practice 2 · Medium] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

[Practice 3 · Hard] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

[Practice 4 · Easy] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

[Practice 5 · Medium] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

📝 Chapter Quiz

Questions Level

Matrices, Linear Layers & Activation Functions

🤖 AI Math Unit 2 — Neural Matrix Operations Applied STEM · Matrix multiplication as linear transformation · Dense layer forward pass y = Wx + b · Dimensions, shapes & batching · Non-linear activations: ReLU, Sigmoid, Softmax · Universal approximation theorem intuition

Deep neural networks are sequences of matrix multiplications alternated with non-linear activation functions. Here we formulate the algebraic forward pass of dense feedforward neural layers.

● Definition 2.1 — Dense Neural Layer Formulation

A fully connected (dense) neural network layer with

n_{\text{in}}

inputs and

n_{\text{out}}

neurons computes an affine linear transformation followed by an element-wise activation

\sigma

\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}, \quad \mathbf{a} = \sigma(\mathbf{z})

where

\mathbf{W} \in \mathbb{R}^{n_{\text{out}} \times n_{\text{in}}}

is the weight matrix,

\mathbf{x} \in \mathbb{R}^{n_{\text{in}}}

is the input vector,

\mathbf{b} \in \mathbb{R}^{n_{\text{out}}}

is the bias vector, and

\mathbf{a} \in \mathbb{R}^{n_{\text{out}}}

is the layer activation vector.

● Deep Conceptual Intuition — Matrices, Linear Layers & Activation Functions

Coordinate grid showing geometric action of a linear transformation mapping △ABC to its image △A'B'C'.

⊙ Explorer · Dense Layer & Two-Neuron ReLU Universal Function Approximator

interactive

Live numerical table for Dense Layer & Two-Neuron ReLU Universal Function Approximator
Parameter / Metric	Value

Neuron 1 Slope $w_{1} =$ 1.5

Neuron 2 Hinge Shift $b_{2} =$ 1

Hinge Activation Kinks

x_{1} = -0.67, x_{2} = 1.00

Output

h(2)

= ReLU

(z_{1})

− ReLU

(z_{2})

h(2) = 2.50

Active Linear Regime2 Active Hinges

See how a hidden layer combining piecewise-linear ReLU activations $\max(0, w_i \cdot x + b_i)$ creates hinged bends capable of approximating nonlinear functions.

Matrices, Linear Layers & Activation Functions — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Matrices, Linear Layers & Activation Functions.

ReLU

\text{ReLU}(z) = \max(0, z)

Sigmoid

\sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1)

Softmax

\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}

Matrix Dimension Rule

(m \times k) \cdot (k \times n) = (m \times n)

◆ Why It Works — First-Principles Derivation (Matrices, Linear Layers & Activation Functions)

\mathbf{u}\cdot\mathbf{v} = c(\mathbf{v}\cdot\mathbf{v}) + 0 \implies c = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{v}\|^2}

Worked Example 2.A — Manual Feedforward Neural Layer

A neural network layer has 2 inputs and 3 neurons with weights and biases:
$\mathbf{W} = \begin{pmatrix} 2 & -1 \\ 0.5 & 3 \\ -1 & 1 \end{pmatrix}, \quad \mathbf{b} = \begin{pmatrix} 1 \\ -2 \\ 0.5 \end{pmatrix}$
For the input $\mathbf{x} = \begin{pmatrix} 3 \\ 2 \end{pmatrix}$ , compute the pre-activation $\mathbf{z}$ and the activated output $\mathbf{a}$ using the ReLU function.

Solution:
Compute $\mathbf{W}\mathbf{x}$ :
$\mathbf{W}\mathbf{x} = \begin{pmatrix} 2(3) + (-1)(2) \\ 0.5(3) + 3(2) \\ -1(3) + 1(2) \end{pmatrix} = \begin{pmatrix} 6 - 2 \\ 1.5 + 6 \\ -3 + 2 \end{pmatrix} = \begin{pmatrix} 4 \\ 7.5 \\ -1 \end{pmatrix}$
Add bias vector $\mathbf{b}$ :
$\mathbf{z} = \begin{pmatrix} 4 \\ 7.5 \\ -1 \end{pmatrix} + \begin{pmatrix} 1 \\ -2 \\ 0.5 \end{pmatrix} = \begin{pmatrix} 5 \\ 5.5 \\ -0.5 \end{pmatrix}$
Apply element-wise $\text{ReLU}(z) = \max(0, z)$ :
$\mathbf{a} = \text{ReLU}(\mathbf{z}) = \begin{pmatrix} \max(0, 5) \\ \max(0, 5.5) \\ \max(0, -0.5) \end{pmatrix} = \begin{pmatrix} 5 \\ 5.5 \\ 0 \end{pmatrix}$ .
Answer: $\mathbf{z} = [5, 5.5, -0.5]^T$ , $\mathbf{a} = [5, 5.5, 0]^T$ .

Worked Example

B

— Matrices, Linear Layers & Activation Functions (Medium Application)

Problem (Medium): In a multiple linear regression model, the Sum of Squared Errors is $\text{SSE} = 100$ and the Total Sum of Squares is $\text{SST} = 500$ . What is the coefficient of determination $R^2$ ?

Step-by-Step Solution: The coefficient of determination measures the proportion of variance explained: $R^2 = 1 - \frac{\text{SSE}}{\text{SST}} = 1 - \frac{100}{500} = 1 - 0.20 = 0.80$ .

Worked Example

C

— Matrices, Linear Layers & Activation Functions (Hard Application)

Problem (Hard): A centered feature matrix $\mathbf{X}$ has shape $N \times 38$ . What is the size and symmetry property of the sample covariance matrix $\mathbf{\Sigma} = \frac{1}{N-1}\mathbf{X}^T\mathbf{X}$ ?

Step-by-Step Solution: Since $\mathbf{X}$ has $38$ columns, $\mathbf{X}^T$ is $38 \times N$ and $\mathbf{X}^T\mathbf{X}$ is $38 \times 38$ . Furthermore, $(\mathbf{X}^T\mathbf{X})^T = \mathbf{X}^T(\mathbf{X}^T)^T = \mathbf{X}^T\mathbf{X}$ , proving it is always symmetric.

Worked Example

D

— Matrices, Linear Layers & Activation Functions (Easy Application)

Problem (Easy): Consider a linear layer with a weight matrix $W$ and an input vector $\mathbf{x}$ . If $W = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}$ and $\mathbf{x} = \begin{pmatrix} 5 \\ 6 \end{pmatrix}$ , compute the output vector $\mathbf{y} = W\mathbf{x}$ .

Step-by-Step Solution: The output vector $\mathbf{y}$ is computed by multiplying the weight matrix $W$ by the input vector $\mathbf{x}$ . This is a standard matrix-vector multiplication. \\ \\ $\mathbf{y} = W\mathbf{x} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 \\ 6 \end{pmatrix}$ \\\ \\ To compute the first element of $\mathbf{y}$ , we take the dot product of the first row of $W$ with $\mathbf{x}$ : $(1)(5) + (2)(6) = 5 + 12 = 17$ .\\ \\ To compute the second element of $\mathbf{y}$ , we take the dot product of the second row of $W$ with $\mathbf{x}$ : $(3)(5) + (4)(6) = 15 + 24 = 39$ .\\ \\ Therefore, the output vector is $\mathbf{y} = \begin{pmatrix} 17 \\ 39 \end{pmatrix}$ .

⚠️ Common Exam Traps — Matrices, Linear Layers & Activation Functions

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Matrices, Linear Layers & Activation Functions)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Computing angle at $B$ using $\overrightarrow{AB}\cdot\overrightarrow{BC}$	Vectors are head-to-tail instead of tail-to-tail, giving $180^\circ - \theta$ .	Reverse $\overrightarrow{AB}$ so both emanate from $B$ : use $\overrightarrow{BA}\cdot\overrightarrow{BC}$ .
Using the same parameter $t$ for two intersecting 3D lines $\mathbf{r}_1(t) = \mathbf{r}_2(t)$	Tests for a simultaneous collision at time $t$ rather than geometric intersection of paths.	Assign distinct parameters $\lambda$ and $\mu$ to the two lines before equating components.
Assuming $(AB)^{-1} = A^{-1}B^{-1}$	Multiplying $(AB)(A^{-1}B^{-1})$ does not cancel inner factors when $AB \ne BA$ .	Use the shoe-sock reversal law: $(AB)^{-1} = B^{-1}A^{-1}$ .

▲ Exam Pacing & Verification Strategy — Matrices, Linear Layers & Activation Functions

[Practice 1 · Medium] A 3D dataset has covariance matrix eigenvalues

\lambda_1 = 62

\lambda_2 = 24

, and

\lambda_3 = 14

. What percentage of total variance is explained by the first principal component

PC_1

💡 Hint 1 of 3

Total variance is

\sum \lambda_i = 62 + 24 + 14

💡 Hint 2 of 3

The proportion explained by

PC_1

is:

\frac{\lambda_1}{\sum \lambda_i}

. Rewrite and substitute:

\frac{\lambda_1}{\sum \lambda_i} = \frac{62}{100}

💡 Hint 3 of 3

Evaluate

\frac{62}{100}

to obtain the final answer.

Total variance is

\sum \lambda_i = 62 + 24 + 14 = 100

. The proportion explained by

PC_1

\frac{\lambda_1}{\sum \lambda_i} = \frac{62}{100} = 62.0\%

[Practice 2 · Hard] Consider a dataset with two features,

X_1

and

X_2

. The covariance matrix for this dataset is given by:

\Sigma = \begin{pmatrix} 10 & 5 \\ 5 & 20 \end{pmatrix}

What is the variance of the first feature,

X_1

💡 Hint 1 of 3

The covariance matrix

\Sigma

for a dataset with two features

X_1

and

X_2

is structured as follows:

\Sigma = \begin{pmatrix} Cov(X_1, X_1) & Cov(X_1, X_2) \\ Cov(X_2, X_1) & Cov(X_2, X_2) \end{pmatrix}

💡 Hint 2 of 3

By definition, the variance of a random variable is the covariance of the variable with itself. Therefore,

Var(X_1) = Cov(X_1, X_1)

and

Var(X_2) = Cov(X_2, X_2)

. In the given covariance matrix:

\Sigma = \begin{pmatrix} 10 & 5 \\ 5 & 20 \end{pmatrix}

💡 Hint 3 of 3

Thus, the variance of

X_1

is 10. The covariance between

X_1

and

X_2

is 5, and the variance of

X_2

is 20.

The covariance matrix

\Sigma

for a dataset with two features

X_1

and

X_2

is structured as follows:

\Sigma = \begin{pmatrix} Cov(X_1, X_1) & Cov(X_1, X_2) \\ Cov(X_2, X_1) & Cov(X_2, X_2) \end{pmatrix}

By definition, the variance of a random variable is the covariance of the variable with itself. Therefore,

Var(X_1) = Cov(X_1, X_1)

and

Var(X_2) = Cov(X_2, X_2)

. In the given covariance matrix:

\Sigma = \begin{pmatrix} 10 & 5 \\ 5 & 20 \end{pmatrix}

The element in the first row and first column represents

Cov(X_1, X_1)

, which is the variance of

X_1

. Thus, the variance of

X_1

is 10. The covariance between

X_1

and

X_2

is 5, and the variance of

X_2

is 20.

[Practice 3 · Easy] Principal Component Analysis (PCA) projects a dataset from

\mathbb{R}^{50}

onto its top

6

principal components. What is the dimension of each transformed data point?

💡 Hint 1 of 3

PCA projection onto

k

eigenvectors transforms each sample vector from

\mathbb{R}^D

\mathbb{R}^k

💡 Hint 2 of 3

Here

k = 6

💡 Hint 3 of 3

So the reduced dimension is

6

PCA projection onto

k

eigenvectors transforms each sample vector from

\mathbb{R}^D

\mathbb{R}^k

. Here

k = 6

, so the reduced dimension is

6

[Practice 4 · Medium] A vector

\mathbf{v}

is transformed into a new vector

\mathbf{v}'

by a linear transformation represented by the matrix

\mathbf{M}

. We are given that

\mathbf{v} = \begin{pmatrix} 2 \ 3 \end{pmatrix}

and

\mathbf{v}' = \begin{pmatrix} 7 \ 11 \end{pmatrix}

. The transformation matrix is of the form

\mathbf{M} = \begin{pmatrix} a & 1 \ 4 & b \end{pmatrix}

. Determine the value of

a

and

b

💡 Hint 1 of 3

The transformation is defined by

\mathbf{v}' = \mathbf{M}\mathbf{v}

. Substituting the given vectors and matrix form:

\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} a & 1 \ 4 & b \end{pmatrix} \begin{pmatrix} 2 \ 3 \end{pmatrix}

💡 Hint 2 of 3

Performing the matrix multiplication:

\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} 2a + 3 \ 8 + 3b \end{pmatrix}

Equating the corresponding components, we get a system of two linear equations: 1)

2a + 3 = 7

💡 Hint 3 of 3

Solving the second equation for

b

3b = 11 - 8

The transformation is defined by

\mathbf{v}' = \mathbf{M}\mathbf{v}

. Substituting the given vectors and matrix form:

\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} a & 1 \ 4 & b \end{pmatrix} \begin{pmatrix} 2 \ 3 \end{pmatrix}

Performing the matrix multiplication:

\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} 2a + 3 \ 8 + 3b \end{pmatrix}

Equating the corresponding components, we get a system of two linear equations: 1)

2a + 3 = 7

8 + 3b = 11

Solving the first equation for

a

2a = 7 - 3

2a = 4

a = 2

Solving the second equation for

b

3b = 11 - 8

3b = 3

b = 1

Thus,

a = 2

and

b = 1

[Practice 5 · Hard] A medical test for a rare disease has a 99\% accuracy rate in detecting the disease when it is present (True Positive Rate) and a 95\% accuracy rate in identifying individuals without the disease (True Negative Rate). If the prevalence of the disease in the population is 0.5\%, what is the probability that a person who tests positive actually has the disease?

💡 Hint 1 of 3

Let D be the event that a person has the disease, and P be the event that the person tests positive.

💡 Hint 2 of 3

We are given: P(P|D) = 0.99 (True Positive Rate) P(not P|not D) = 0.95 (True Negative Rate), which means P(P|not D) = 1 - 0.95 = 0.05 (False Positive Rate).

💡 Hint 3 of 3

Converting to a percentage, this is approximately 9.05\%, which rounds to 9.1\%.

Let D be the event that a person has the disease, and P be the event that the person tests positive. We are given: P(P|D) = 0.99 (True Positive Rate) P(not P|not D) = 0.95 (True Negative Rate), which means P(P|not D) = 1 - 0.95 = 0.05 (False Positive Rate) P(D) = 0.005 (Prevalence of the disease) From P(D), we know P(not D) = 1 - P(D) = 1 - 0.005 = 0.995. We want to find P(D|P), the probability that a person has the disease given a positive test result. Using Bayes' Theorem: P(D|P) = [P(P|D) * P(D)] / P(P) To find P(P), we use the law of total probability: P(P) = P(P|D) * P(D) + P(P|not D) * P(not D) P(P) = (0.99 * 0.005) + (0.05 * 0.995) P(P) = 0.00495 + 0.04975 P(P) = 0.0547 Now, substitute this back into Bayes' Theorem: P(D|P) = (0.99 * 0.005) / 0.0547 P(D|P) = 0.00495 / 0.0547 P(D|P) \approx 0.09049 Converting to a percentage, this is approximately 9.05\%, which rounds to 9.1\%.

📝 Chapter Quiz

Questions Level

Loss Functions, Gradients & Optimization

🤖 AI Math Unit 3 — Gradient Descent Optimization Applied STEM · Loss functions: Mean Squared Error (MSE) · Partial derivatives & the gradient vector ∇L · Learning rate η and convergence · Gradient descent update rule w ← w - η ∇L · Convex vs non-convex loss landscapes

Machine learning models learn by minimizing a loss function via gradient descent. Calculus provides the directional derivative pointing toward the steepest descent in parameter space.

▲ Theorem 3.1 — Gradient Descent Update Equation

Let

L(\mathbf{w})

be a differentiable loss function over weight parameters

\mathbf{w}

. The gradient

\nabla L(\mathbf{w}) = \left[ \frac{\partial L}{\partial w_1}, \dots, \frac{\partial L}{\partial w_d} \right]^T

points in the direction of greatest rate of increase.
To minimize loss, parameter updates step in the opposite direction scaled by learning rate

\eta > 0

\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla L(\mathbf{w}_t)

● Deep Conceptual Intuition — Loss Functions, Gradients & Optimization

Proportional Invariance, Linear Structure & Quantitative Modeling: Across Mathematics for Artificial Intelligence & Machine Learning, multi-step problems reduce to identifying invariant quantities—constant rates of change $m = \dfrac{\Delta y}{\Delta x}$ , conservation of total work/mass, or linear system rank conditions. A two-variable linear system $a_1 x + b_1 y = c_1$ , $a_2 x + b_2 y = c_2$ has a unique intersection if $\dfrac{a_1}{a_2} \ne \dfrac{b_1}{b_2}$ , no solution (parallel lines) if $\dfrac{a_1}{a_2} = \dfrac{b_1}{b_2} \ne \dfrac{c_1}{c_2}$ , and infinitely many solutions if all three ratios coincide.

Coordinate intersection of linear system

y = 2x + 1

and

y = -x + 7

at unique equilibrium point

(2, 5)

⊙ Explorer · Gradient Descent Optimization w_{t+1} =

w_t - \eta

·∇

L(w_t)

interactive

Live numerical table for Gradient Descent Optimization w_{t+1} = $w_t - \eta$ ·∇ $L(w_t)$
Parameter / Metric	Value

Learning Rate $\eta =$ 0.35

Initial Weight $w_{0} =$ -2.5

Initial Gradient ∇

L(w_{0})

∇L(w_{0}) = -7.00

Weight

w_{6}

after 6 Steps

w_{6} = 0.997

Convergence RegimeMonotone Convergent

Trace 6 steps of gradient descent w_{t+1} = $w_t - \eta \cdot 2(w_t - 1)$ on the convex loss bowl $L(w) = (w - 1)^{2} + 0.5$ . Notice how $\eta > 0.5$ overshoots the minimum and $\eta \ge 1.0$ diverges!

Loss Functions, Gradients & Optimization — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Loss Functions, Gradients & Optimization.

MSE Loss

\text{MSE} = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2

Gradient Vector

\nabla L = \left[ \frac{\partial L}{\partial w}, \frac{\partial L}{\partial b} \right]^T

Learning Rate

Too high: divergence/oscillation. Too low: slow convergence.

Stochastic GD

Updates weights on mini-batches of size

B \ll N

◆ Why It Works — First-Principles Derivation (Loss Functions, Gradients & Optimization)

Why Successive Multipliers & Harmonic Average Speed Work: If a quantity $Q_0$ undergoes a percentage change $p\%$ followed by $q\%$ , the compound multiplier is the product of individual scale factors:

Q_{\text{final}} = Q_0\left(1 + \frac{p}{100}\right)\!\left(1 + \frac{q}{100}\right) = Q_0\left(1 + \frac{p + q}{100} + \frac{pq}{10000}\right)

Notice the cross term $\dfrac{pq}{10000}$ : an increase of $p\%$ followed by a decrease of $p\%$ yields $Q_0\bigl(1 - (p/100)^2\bigr) < Q_0$. Similarly, traveling equal distance $d$ at speeds $v_1$ and $v_2$ takes total time $T = \frac{d}{v_1} + \frac{d}{v_2}$, so average speed is the harmonic mean $v_{\text{avg}} = \dfrac{2d}{d/v_1 + d/v_2} = \dfrac{2v_1 v_2}{v_1 + v_2}$.

Worked Example 3.A — Single Step of Gradient Descent

Consider a simple linear model $\hat{y} = wx$ with squared loss for a single data point $(x, y) = (2, 8)$ :
$L(w) = (y - wx)^2 = (8 - 2w)^2$
If the current weight is $w_0 = 1.0$ and the learning rate is $\eta = 0.05$ , compute the gradient $\frac{dL}{dw}$ and the updated weight $w_1$ .

Solution:
Apply the chain rule to differentiate $L(w)$ :
$\frac{dL}{dw} = 2(8 - 2w) \cdot (-2) = -4(8 - 2w) = -32 + 8w$
Evaluate at $w_0 = 1.0$ :
$\left.\frac{dL}{dw}\right|_{w=1.0} = -32 + 8(1) = -24$
Apply gradient descent update:
$w_1 = w_0 - \eta \left(\frac{dL}{dw}\right) = 1.0 - (0.05)(-24) = 1.0 + 1.2 = 2.2$ .
Notice that with $w_0 = 1.0$ , prediction $\hat{y} = 2$ was too low (target was 8). The gradient step increased $w$ to 2.2, giving $\hat{y} = 4.4$ , moving closer to 8.
Answer: Gradient = $-24$ ; updated weight $w_1 = 2.2$ .

Worked Example

B

— Loss Functions, Gradients & Optimization (Medium Application)

Problem (Medium): Consider a univariate regression model where the predicted output $\hat{y}$ is given by $\hat{y} = ax + b$ . The loss function is the Mean Squared Error (MSE), defined as $L = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ . For a single data point $(x_1, y_1) = (2, 5)$ , the loss is $L = (5 - (2a + b))^2$ . The gradient of the loss function with respect to $a$ is $\frac{\partial L}{\partial a} = -2(5 - (2a + b)) \cdot 2$ . If during gradient descent, with a learning rate of $\eta = 0.1$ , the parameters $a$ and $b$ are updated from $a_{old} = 1$ and $b_{old} = 1$ , to $a_{new} = 0.8$ and $b_{new} = 0.8$ , what is the value of $b$ that would have resulted in $\frac{\partial L}{\partial b} = 0$ for this single data point, assuming $a=1$ and $b=1$ were the values *before* the gradient calculation that led to the update to $a_{new}=0.8, b_{new}=0.8$ ?

Step-by-Step Solution: The loss function for a single data point is $L = (y_1 - (ax_1 + b))^2$ . Given $(x_1, y_1) = (2, 5)$ , so $L = (5 - (2a + b))^2$ . The gradient of the loss with respect to $b$ is $\frac{\partial L}{\partial b} = 2(5 - (2a + b)) \cdot (-1) = -2(5 - 2a - b)$ . We are asked to find the value of $b$ for which $\frac{\partial L}{\partial b} = 0$ , assuming $a=1$ and $b=1$ were the parameters *before* the update step that led to $a_{new}=0.8, b_{new}=0.8$ . This means we evaluate the gradient at $a=1$ and the *unknown* value of $b$ . Setting $\frac{\partial L}{\partial b} = 0$ : $-2(5 - 2(1) - b) = 0$ $5 - 2 - b = 0$ $3 - b = 0$ $b = 3$ . The provided gradient update information $\frac{\partial L}{\partial a} = -2(5 - (2a + b)) \cdot 2$ and the learning rate $\eta = 0.1$ are distractors. The question specifically asks for the value of $b$ that makes $\frac{\partial L}{\partial b} = 0$ *at the point where $a=1$ *. The update rule to $a_{new}=0.8, b_{new}=0.8$ implies that the gradient of $a$ was $\frac{a_{old} - a_{new}}{\eta} = \frac{1 - 0.8}{0.1} = 2$ , and the gradient of $b$ was $\frac{b_{old} - b_{new}}{\eta} = \frac{1 - 0.8}{0.1} = 2$ . Let's verify: If $a=1$ and $b=3$ , then $\frac{\partial L}{\partial b} = -2(5 - 2(1) - 3) = -2(5 - 2 - 3) = -2(0) = 0$ . If $a=1$ and $b=3$ , $\frac{\partial L}{\partial a} = -2(5 - 2(1) - 3) \cdot 2 = -2(0) \cdot 2 = 0$ . This scenario implies that the point $a=1, b=3$ is a minimum for the loss function, but the question is about finding the value of $b$ that makes the gradient of $b$ zero at $a=1$ , which is $b=3$ . The provided update $a_{new}=0.8, b_{new}=0.8$ from $a_{old}=1, b_{old}=1$ is a different step in the optimization process and is not directly used to find the value of $b$ that makes the gradient of $b$ zero at $a=1$ .

مسألة (Medium): Consider a univariate regression model where the predicted output $\hat{y}$ is given by $\hat{y} = ax + b$ . The loss function is the Mean Squared Error (MSE), defined as $L = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ . For a single data point $(x_1, y_1) = (2, 5)$ , the loss is $L = (5 - (2a + b))^2$ . The gradient of the loss function with respect to $a$ is $\frac{\partial L}{\partial a} = -2(5 - (2a + b)) \cdot 2$ . If during gradient descent, with a learning rate of $\eta = 0.1$ , the parameters $a$ and $b$ are updated from $a_{old} = 1$ and $b_{old} = 1$ , to $a_{new} = 0.8$ and $b_{new} = 0.8$ , what is the value of $b$ that would have resulted in $\frac{\partial L}{\partial b} = 0$ for this single data point, assuming $a=1$ and $b=1$ were the values *before* the gradient calculation that led to the update to $a_{new}=0.8, b_{new}=0.8$ ?

الحل خطوة بخطوة: The loss function for a single data point is $L = (y_1 - (ax_1 + b))^2$ . Given $(x_1, y_1) = (2, 5)$ , so $L = (5 - (2a + b))^2$ . The gradient of the loss with respect to $b$ is $\frac{\partial L}{\partial b} = 2(5 - (2a + b)) \cdot (-1) = -2(5 - 2a - b)$ . We are asked to find the value of $b$ for which $\frac{\partial L}{\partial b} = 0$ , assuming $a=1$ and $b=1$ were the parameters *before* the update step that led to $a_{new}=0.8, b_{new}=0.8$ . This means we evaluate the gradient at $a=1$ and the *unknown* value of $b$ . Setting $\frac{\partial L}{\partial b} = 0$ : $-2(5 - 2(1) - b) = 0$ $5 - 2 - b = 0$ $3 - b = 0$ $b = 3$ . The provided gradient update information $\frac{\partial L}{\partial a} = -2(5 - (2a + b)) \cdot 2$ and the learning rate $\eta = 0.1$ are distractors. The question specifically asks for the value of $b$ that makes $\frac{\partial L}{\partial b} = 0$ *at the point where $a=1$ *. The update rule to $a_{new}=0.8, b_{new}=0.8$ implies that the gradient of $a$ was $\frac{a_{old} - a_{new}}{\eta} = \frac{1 - 0.8}{0.1} = 2$ , and the gradient of $b$ was $\frac{b_{old} - b_{new}}{\eta} = \frac{1 - 0.8}{0.1} = 2$ . Let's verify: If $a=1$ and $b=3$ , then $\frac{\partial L}{\partial b} = -2(5 - 2(1) - 3) = -2(5 - 2 - 3) = -2(0) = 0$ . If $a=1$ and $b=3$ , $\frac{\partial L}{\partial a} = -2(5 - 2(1) - 3) \cdot 2 = -2(0) \cdot 2 = 0$ . This scenario implies that the point $a=1, b=3$ is a minimum for the loss function, but the question is about finding the value of $b$ that makes the gradient of $b$ zero at $a=1$ , which is $b=3$ . The provided update $a_{new}=0.8, b_{new}=0.8$ from $a_{old}=1, b_{old}=1$ is a different step in the optimization process and is not directly used to find the value of $b$ that makes the gradient of $b$ zero at $a=1$ .

Worked Example

C

— Loss Functions, Gradients & Optimization (Hard Application)

Problem (Hard): Consider the following quadratic loss function for a regression problem: $L(y, \hat{y}) = (y - \hat{y})^2$ . If the predicted value $\hat{y} = 5$ and the true value $y = 3$ , what is the value of the loss function?

Step-by-Step Solution: The quadratic loss function is defined as $L(y, \hat{y}) = (y - \hat{y})^2$ . Given the true value $y = 3$ and the predicted value $\hat{y} = 5$ , we substitute these values into the loss function: $L(3, 5) = (3 - 5)^2$ $L(3, 5) = (-2)^2$ $L(3, 5) = 4$ Therefore, the value of the loss function is 4.

Worked Example

D

— Loss Functions, Gradients & Optimization (Easy Application)

Problem (Easy): Consider the quadratic loss function $L(w) = w^2 - 4w + 3$ . Find the value of $w$ where the function reaches its minimum.

Step-by-Step Solution: The given loss function is a quadratic function $L(w) = w^2 - 4w + 3$ . This represents a parabola opening upwards. The minimum value of a quadratic function of the form $aw^2 + bw + c$ occurs at $w = -\frac{b}{2a}$ . In this case, $a = 1$ and $b = -4$ . Therefore, the minimum occurs at $w = -\frac{-4}{2(1)} = \frac{4}{2} = 2$ . Alternatively, we can find the minimum by taking the derivative and setting it to zero: $L'(w) = 2w - 4$ . Setting $L'(w) = 0$ gives $2w - 4 = 0$ , which implies $2w = 4$ , so $w = 2$ .

⚠️ Common Exam Traps — Loss Functions, Gradients & Optimization

1. Adding Successive Percentages or Averaging Speeds Arithmetically: On Mathematics for Artificial Intelligence & Machine Learning, a $20\%$ discount followed by a $20\%$ increase gives $0.80 \times 1.20 = 0.96$ (a net $4\%$ loss, not $0\%$ ). Likewise, average speed for a round trip at $40$ and $60\text{ km/h}$ is $\dfrac{2(40)(60)}{100} = 48\text{ km/h}$ , never $50\text{ km/h}$ .

2. Failing to Flip the Inequality Sign When Multiplying/Dividing by a Negative: In $-3x + 5 \le 8$ , dividing by $-3$ strictly reverses $\le$ to $\ge$ .

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Loss Functions, Gradients & Optimization)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Claiming $+25\%$ followed by $-25\%$ returns to $100\%$	The $25\%$ reduction applies to the larger base $1.25 Q_0$ , not $Q_0$ .	Multiply scale factors: $1.25 \times 0.75 = 0.9375$ (a $6.25\%$ net decrease).
Solving $-3x > 12$ as $x > -4$	Dividing by a negative number reverses order on the real number line.	Flip the inequality symbol whenever multiplying or dividing by a negative: $x < -4$.
Computing round-trip average speed as $\frac{v_1 + v_2}{2}$	Time spent at the slower speed is longer than time spent at the faster speed.	Use total distance over total time: $v_{\text{avg}} = \frac{2v_1 v_2}{v_1 + v_2}$ .

▲ Exam Pacing & Verification Strategy — Loss Functions, Gradients & Optimization

High-Speed Exam Shortcut (Mathematics for Artificial Intelligence & Machine Learning): When a linear system asks for a combined expression like $3x + 3y$ rather than $x$ and $y$ individually, add or subtract the two equations directly before isolating either variable—this often yields the target expression in a single step.

[Practice 1 · Medium] Consider the following quadratic loss function:

L(w) = 3w^2 - 12w + 10

. What is the value of

w

that minimizes this loss function?

💡 Hint 1 of 3

To find the minimum of the quadratic loss function

L(w) = 3w^2 - 12w + 10

, we can find the derivative with respect to

w

and set it to zero.

💡 Hint 2 of 3

The derivative of

L(w)

with respect to

w

is:

\frac{dL}{dw} = 6w - 12

. Setting the derivative to zero to find the critical point:

6w - 12 = 0

6w = 12

w = \frac{12}{6}

w = 2

💡 Hint 3 of 3

The second derivative is

\frac{d^2L}{dw^2} = 6

, which is positive, confirming that

w

corresponds to a minimum.

To find the minimum of the quadratic loss function

L(w) = 3w^2 - 12w + 10

, we can find the derivative with respect to

w

and set it to zero. The derivative of

L(w)

with respect to

w

is:

\frac{dL}{dw} = 6w - 12

. Setting the derivative to zero to find the critical point:

6w - 12 = 0

6w = 12

w = \frac{12}{6}

w = 2

. The second derivative is

\frac{d^2L}{dw^2} = 6

, which is positive, confirming that

w=2

corresponds to a minimum.

[Practice 2 · Hard] Evaluate the output of the standard ReLU activation function

f(z) = \max(0, z)

for inputs

z_1 = -4

and

z_2 = 6

💡 Hint 1 of 3

By definition of the Rectified Linear Unit:

\text{ReLU}(z) = z

z > 0

💡 Hint 2 of 3

And

0

z \le 0

💡 Hint 3 of 3

Thus

f(-4) = 0

and

f(6)

By definition of the Rectified Linear Unit:

\text{ReLU}(z) = z

z > 0

, and

0

z \le 0

. Thus

f(-4) = 0

and

f(6) = 6

[Practice 3 · Easy] In gradient descent, a weight currently has value

w = 60

. If the gradient of the loss is

\frac{\partial L}{\partial w} = 24

and the learning rate is

\eta = 0.6

, find the updated weight

w'

💡 Hint 1 of 3

Using the gradient descent update rule:

w' = w - \eta \frac{\partial L}{\partial w}

💡 Hint 2 of 3

Substitute and simplify:

w' = 60 - (0.6)(24)

💡 Hint 3 of 3

Compute the final value from:

60 - 14.40

Using the gradient descent update rule:

w' = w - \eta \frac{\partial L}{\partial w} = 60 - (0.6)(24) = 60 - 14.40 = 45.60

[Practice 4 · Medium] Consider the scalar loss function

L(w) = 5w^2 - 8w + 5

. Find the analytical gradient

\nabla L(w)

evaluated at

w = 8

💡 Hint 1 of 3

Differentiate with respect to

w

\frac{dL}{dw} = 2(5)w - 8 = 10w - 8

💡 Hint 2 of 3

w = 8

\nabla L = 10(8) - 8

. Substitute and simplify:

\nabla L = 80 - 8

💡 Hint 3 of 3

Compute the final value from:

80 - 8

Differentiate with respect to

w

\frac{dL}{dw} = 2(5)w - 8 = 10w - 8

. At

w = 8

\nabla L = 10(8) - 8 = 80 - 8 = 72

[Practice 5 · Hard] Two document embeddings

\mathbf{a} = [6, 12]^T

and

\mathbf{b} = [12, 6]^T

are compared. If

\|\mathbf{a}\| = \|\mathbf{b}\| = \sqrt{180}

, calculate their exact cosine similarity.

💡 Hint 1 of 3

Dot product

\mathbf{a} \cdot \mathbf{b} = (6)(12) + (12)(6)

💡 Hint 2 of 3

Product of norms

\|\mathbf{a}\| \|\mathbf{b}\| = 180

💡 Hint 3 of 3

Cosine similarity =

\frac{144}{180}

Dot product

\mathbf{a} \cdot \mathbf{b} = (6)(12) + (12)(6) = 144

. Product of norms

\|\mathbf{a}\| \|\mathbf{b}\| = 180

. Cosine similarity =

\frac{144}{180} = 0.800

📝 Chapter Quiz

Questions Level

Conditional Probability, Bayes' Rule & Classifiers

🤖 AI Math Unit 4 — Probabilistic Machine Learning Applied STEM · Joint and conditional probability P(A|B) · Bayes' Theorem: Prior, Likelihood, Posterior · Total probability theorem · Naive Bayes independence assumption · Text spam filtering & medical diagnostics

Uncertainty is omnipresent in real-world data. Bayes' Theorem provides the mathematical framework for updating beliefs when observing new evidence, forming the core of Bayesian machine learning and Naive Bayes classifiers.

▲ Theorem 4.1 — Bayes' Theorem in Machine Learning

For hypothesis (class)

C

and observed evidence (features)

\mathbf{x}

P(C | \mathbf{x}) = \frac{P(\mathbf{x} | C) P(C)}{P(\mathbf{x})}

Where:
•

P(C)

is the Prior probability of class

C

.
•

P(\mathbf{x} | C)

is the Likelihood of observing features

\mathbf{x}

given class

C

.
•

P(\mathbf{x}) = \sum_k P(\mathbf{x}|C_k)P(C_k)

is the Marginal probability (Evidence).
•

P(C | \mathbf{x})

is the Posterior probability.

● Deep Conceptual Intuition — Conditional Probability, Bayes' Rule & Classifiers

Coordinate intersection of linear system

y = 2x + 1

and

y = -x + 7

at unique equilibrium point

(2, 5)

⊙ Explorer · Bayes' Theorem Classifier: Prior Base Rate vs. Posterior

P(C_{1} | +)

interactive

Live numerical table for Bayes' Theorem Classifier: Prior Base Rate vs. Posterior $P(C_{1} | +)$
Parameter / Metric	Value

Prior Base Rate $P(C_{1}) =$ 0.2

False Positive Rate $P(+ | C_{0}) =$ 0.15

Evidence Marginal

P(+)

P(+) = 0.300

Posterior Precision

P(C_{1} | +)

P(C_{1}|+) = 60.0\%

Bayes Lift

P(C_{1}|+) / P(C_{1})

3.00× Lift

With fixed sensitivity $P(+ | C_{1}) = 0.90$ , see how a low base rate $P(C_{1})$ or high false-positive rate $P(+ | C_{0})$ dramatically reduces the posterior probability $P(C_{1} | +)$ .

Conditional Probability, Bayes' Rule & Classifiers — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Conditional Probability, Bayes' Rule & Classifiers.

Conditional Probability

P(A|B) = \frac{P(A \cap B)}{P(B)}

Naive Bayes Assumption

P(\mathbf{x}|C) = \prod_{j=1}^d P(x_j|C)

(conditional independence)

Decision Rule

\hat{C} = \arg\max_C P(C) \prod_{j=1}^d P(x_j|C)

Laplace Smoothing

Prevents zero probability:

\frac{\text{count} + 1}{\text{total} + |V|}

◆ Why It Works — First-Principles Derivation (Conditional Probability, Bayes' Rule & Classifiers)

Q_{\text{final}} = Q_0\left(1 + \frac{p}{100}\right)\!\left(1 + \frac{q}{100}\right) = Q_0\left(1 + \frac{p + q}{100} + \frac{pq}{10000}\right)

Worked Example 4.A — Bayesian Spam Classification

An email filter processes emails where $20\%$ are spam ( $P(S) = 0.20$ ) and $80\%$ are legitimate ( $P(L) = 0.80$ ). The word 'lottery' appears in $60\%$ of spam emails ( $P(W|S) = 0.60$ ) and in $1\%$ of legitimate emails ( $P(W|L) = 0.01$ ). If an incoming email contains the word 'lottery', what is the probability that it is spam?

Solution:
By Bayes' Theorem:
$P(S|W) = \frac{P(W|S)P(S)}{P(W)}$
Calculate total probability $P(W)$ :
$P(W) = P(W|S)P(S) + P(W|L)P(L) = (0.60)(0.20) + (0.01)(0.80) = 0.12 + 0.008 = 0.128$
Compute posterior:
$P(S|W) = \frac{0.12}{0.128} = \frac{120}{128} = \frac{15}{16} = 0.9375$ .
Despite only 20% of all emails being spam initially, observing the word 'lottery' raises the probability to $93.75\%$ .
Answer: $93.75\%$ (or $0.938$ ).

Worked Example

B

— Conditional Probability, Bayes' Rule & Classifiers (Medium Application)

Problem (Medium): Given $n = 4$ independent observations from a normal distribution with sum $\sum_{i=1}^{4} x_i = 12$ , what is the Maximum Likelihood Estimate (MLE) $\hat{\mu}_{\text{MLE}}$ ?

Step-by-Step Solution: The log-likelihood $\ln L(\mu) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum (x_i - \mu)^2$ . Setting $\frac{\partial \ln L}{\partial \mu} = 0$ yields the sample mean $\hat{\mu} = \frac{1}{n}\sum x_i = \frac{12}{4} = 3$ .

Worked Example

C

— Conditional Probability, Bayes' Rule & Classifiers (Hard Application)

Problem (Hard): A logistic regression model produces a class probability $p = P(y=1|\mathbf{x}) = 0.40$ . Compute the corresponding log-odds (logit) $\ln\left(\frac{p}{1-p}\right)$ .

Step-by-Step Solution: Odds are $\frac{p}{1-p} = \frac{0.40}{0.60} = 0.6667$ . Taking natural logarithm: $\ln(0.6667) = -0.405$ .

Worked Example

D

— Conditional Probability, Bayes' Rule & Classifiers (Easy Application)

Problem (Easy): A medical diagnostic test for a rare disease is designed to be highly sensitive and specific. The disease affects $1$ in every $1000$ individuals in the population. The test has a sensitivity of $99$ \% (meaning it correctly identifies $99$ \% of individuals who have the disease) and a specificity of $95$ \% (meaning it correctly identifies $95$ \% of individuals who do not have the disease). If a randomly selected individual from the population tests positive, what is the probability that they actually have the disease?

Step-by-Step Solution: Let D be the event that an individual has the disease, and P be the event that the individual tests positive. We are given the following probabilities: - Prior probability of having the disease: $P(D) = \frac{1}{1000} = 0.001$ - Prior probability of not having the disease: $P(D^c) = 1 - P(D) = 1 - 0.001 = 0.999$ - Sensitivity (true positive rate): $P(P|D) = 0.99$ - Specificity (true negative rate): $P(P^c|D^c) = 0.95$ From specificity, we can derive the false positive rate: $P(P|D^c) = 1 - P(P^c|D^c) = 1 - 0.95 = 0.05$ . We want to find the probability that an individual actually has the disease given a positive test result, which is $P(D|P)$ . We can use Bayes' Theorem: $P(D|P) = \frac{P(P|D) P(D)}{P(P)}$ To find $P(P)$ , the overall probability of testing positive, we can use the law of total probability: $P(P) = P(P|D) P(D) + P(P|D^c) P(D^c)$ Substitute the given values: $P(P) = (0.99)(0.001) + (0.05)(0.999)$ $P(P) = 0.00099 + 0.04995$ $P(P) = 0.05094$ Now, substitute $P(P)$ back into Bayes' Theorem: $P(D|P) = \frac{(0.99)(0.001)}{0.05094}$ $P(D|P) = \frac{0.00099}{0.05094}$ $P(D|P) \approx 0.0194346$ Therefore, the probability that a person who tests positive actually has the disease is approximately $0.0194$ , which is closest to $0.019$ .

مسألة (Easy): A medical diagnostic test for a rare disease is designed to be highly sensitive and specific. The disease affects $1$ in every $1000$ individuals in the population. The test has a sensitivity of $99$ \% (meaning it correctly identifies $99$ \% of individuals who have the disease) and a specificity of $95$ \% (meaning it correctly identifies $95$ \% of individuals who do not have the disease). If a randomly selected individual from the population tests positive, what is the probability that they actually have the disease?

الحل خطوة بخطوة: Let D be the event that an individual has the disease, and P be the event that the individual tests positive. We are given the following probabilities: - Prior probability of having the disease: $P(D) = \frac{1}{1000} = 0.001$ - Prior probability of not having the disease: $P(D^c) = 1 - P(D) = 1 - 0.001 = 0.999$ - Sensitivity (true positive rate): $P(P|D) = 0.99$ - Specificity (true negative rate): $P(P^c|D^c) = 0.95$ From specificity, we can derive the false positive rate: $P(P|D^c) = 1 - P(P^c|D^c) = 1 - 0.95 = 0.05$ . We want to find the probability that an individual actually has the disease given a positive test result, which is $P(D|P)$ . We can use Bayes' Theorem: $P(D|P) = \frac{P(P|D) P(D)}{P(P)}$ To find $P(P)$ , the overall probability of testing positive, we can use the law of total probability: $P(P) = P(P|D) P(D) + P(P|D^c) P(D^c)$ Substitute the given values: $P(P) = (0.99)(0.001) + (0.05)(0.999)$ $P(P) = 0.00099 + 0.04995$ $P(P) = 0.05094$ Now, substitute $P(P)$ back into Bayes' Theorem: $P(D|P) = \frac{(0.99)(0.001)}{0.05094}$ $P(D|P) = \frac{0.00099}{0.05094}$ $P(D|P) \approx 0.0194346$ Therefore, the probability that a person who tests positive actually has the disease is approximately $0.0194$ , which is closest to $0.019$ .

⚠️ Common Exam Traps — Conditional Probability, Bayes' Rule & Classifiers

2. Failing to Flip the Inequality Sign When Multiplying/Dividing by a Negative: In $-5x + 7 \le 12$ , dividing by $-5$ strictly reverses $\le$ to $\ge$ .

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Conditional Probability, Bayes' Rule & Classifiers)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Claiming $+25\%$ followed by $-25\%$ returns to $100\%$	The $25\%$ reduction applies to the larger base $1.25 Q_0$ , not $Q_0$ .	Multiply scale factors: $1.25 \times 0.75 = 0.9375$ (a $6.25\%$ net decrease).
Solving $-3x > 12$ as $x > -4$	Dividing by a negative number reverses order on the real number line.	Flip the inequality symbol whenever multiplying or dividing by a negative: $x < -4$.
Computing round-trip average speed as $\frac{v_1 + v_2}{2}$	Time spent at the slower speed is longer than time spent at the faster speed.	Use total distance over total time: $v_{\text{avg}} = \frac{2v_1 v_2}{v_1 + v_2}$ .

▲ Exam Pacing & Verification Strategy — Conditional Probability, Bayes' Rule & Classifiers

[Practice 1 · Medium] A spam classification model identifies

TP = 30

genuine spam emails and

FP = 10

legitimate emails falsely flagged as spam. What is the precision of this model?

💡 Hint 1 of 3

Precision is the fraction of predicted positive instances that are truly positive:

\text{Precision} = \frac{TP}{TP + FP}

💡 Hint 2 of 3

Substitute and simplify:

\text{Precision} = \frac{30}{30 + 10}

💡 Hint 3 of 3

Compute the final value from:

\frac{30}{30 + 10}

Precision is the fraction of predicted positive instances that are truly positive:

\text{Precision} = \frac{TP}{TP + FP} = \frac{30}{30 + 10} = \frac{30}{40} = 75.0\%

[Practice 2 · Hard] Compute the Manhattan (

L_1

) norm and Euclidean (

L_2

) norm of the word vector

\mathbf{w} = [10, -15]^T

💡 Hint 1 of 3

Set up the initial relation:

L_1

norm is the sum of absolute values:

|10| + |-15| = 25

💡 Hint 2 of 3

Rewrite and simplify to obtain:

L_2

norm is the square root of sum of squares:

\sqrt{(10)^2 + (-15)^2}

. Rewrite and substitute:

\sqrt{(10)^2 + (-15)^2} = \sqrt{100 + 225}

💡 Hint 3 of 3

Evaluate

\sqrt{100 + 225}

to obtain the final answer.

L_1

norm is the sum of absolute values:

|10| + |-15| = 25

L_2

norm is the square root of sum of squares:

\sqrt{(10)^2 + (-15)^2} = \sqrt{100 + 225} = \sqrt{325}

[Practice 3 · Easy] Consider two vectors,

\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}

and

\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}

. What is the vector representing

2\mathbf{u} - \mathbf{v}

💡 Hint 1 of 3

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

💡 Hint 2 of 3

Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix}

💡 Hint 3 of 3

Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

To find the vector

2\mathbf{u} - \mathbf{v}

, we first compute

2\mathbf{u}

and then subtract

\mathbf{v}

from the result.

2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}

. Now, we subtract

\mathbf{v}

from

2\mathbf{u}

2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}

. Therefore, the resulting vector is

\begin{bmatrix} 5 \\ 10 \end{bmatrix}

[Practice 4 · Medium] Consider a dataset of

n

observations

(x_i, y_i)

for

i = 1, \dots, n

, where

x_i

are scalar predictor variables and

y_i

are scalar response variables. The ordinary least squares (OLS) estimates for the intercept

\beta_0

and the slope

\beta_1

of the linear model

y = \beta_0 + \beta_1 x + \epsilon

are given by

\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}

and

\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}

. Suppose we have a dataset where

n = 5

\sum x_i = 15

\sum y_i = 30

\sum x_i^2 = 55

\sum y_i^2 = 210

, and

\sum x_i y_i = 105

. A new data point

(x_6, y_6)

is added to the dataset. If the new regression line that minimizes the sum of squared errors for the

n+1 = 6

points is parallel to the original regression line (i.e.,

\hat{\beta}_1^{\text{new}} = \hat{\beta}_1^{\text{old}}

), what is the value of

y_6

in terms of

x_6

💡 Hint 1 of 3

For the original

n = 5

observations,

\bar{x} = \frac{15}{5} = 3

and

\bar{y} = \frac{30}{5}

💡 Hint 2 of 3

The sums of squares are

S_{xx} = 55 - 5(3^2) = 10

and

S_{xy} = 105 - 5(3)(6) = 15

, giving slope

\hat{\beta}_1 = \frac{15}{10} = 1.5

and intercept

\hat{\beta}_0 = 6 - 1.5(3)

💡 Hint 3 of 3

When a 6th point

(x_6, y_6)

lying on the existing regression line

y_6 = 1.5x_6 + 1.5

is added, the residual is zero and the OLS slope remains unchanged at

1.5

For the original

n = 5

observations,

\bar{x} = \frac{15}{5} = 3

and

\bar{y} = \frac{30}{5} = 6

. The sums of squares are

S_{xx} = 55 - 5(3^2) = 10

and

S_{xy} = 105 - 5(3)(6) = 15

, giving slope

\hat{\beta}_1 = \frac{15}{10} = 1.5

and intercept

\hat{\beta}_0 = 6 - 1.5(3) = 1.5

. When a 6th point

(x_6, y_6)

lying on the existing regression line

y_6 = 1.5x_6 + 1.5

is added, the residual is zero and the OLS slope remains unchanged at

1.5

[Practice 5 · Hard] Consider a dataset of

n

data points, where each data point is a vector in

\mathbb{R}^{D}

. A non-linear dimensionality reduction technique maps these data points to a lower-dimensional space

\mathbb{R}^{d}

. The mapping function for a single data point

\mathbf{x} \in \mathbb{R}^{D}

is given by

f(\mathbf{x}) = \phi(\mathbf{x})

, where

\phi: \mathbb{R}^{D} \to \mathbb{R}^{d}

is a function designed to preserve local neighborhood structures. Suppose the mapping

\phi

is defined by an explicit feature map

\phi(\mathbf{x}) = [\mathbf{x}^T \mathbf{x}, \mathbf{x}^T \mathbf{x}, \dots, \mathbf{x}^T \mathbf{x}]^T

where the vector

\mathbf{x} \in \mathbb{R}^{10}

and

d

is the dimension of the embedded space. If the goal is to embed these points into a space such that pairwise Euclidean distances are preserved as closely as possible under a specific distortion, and it is found that

d=15

yields optimal results for this mapping under a certain constraint, determine the effective rank of the feature map's image space.

💡 Hint 1 of 3

The feature map is defined as

\phi(\mathbf{x}) = [\mathbf{x}^T \mathbf{x}, \mathbf{x}^T \mathbf{x}, \dots, \mathbf{x}^T \mathbf{x}]^T

, where

\mathbf{x} \in \mathbb{R}^{10}

💡 Hint 2 of 3

The term

\mathbf{x}^T \mathbf{x}

is the squared Euclidean norm of

\mathbf{x}

, denoted as

\|\mathbf{x}\|^2

. This is a scalar value.

💡 Hint 3 of 3

The problem states that

d

, but the effective rank of the image of this specific feature map is always 1, as all output vectors lie on a single line passing through the origin.

The feature map is defined as

\phi(\mathbf{x}) = [\mathbf{x}^T \mathbf{x}, \mathbf{x}^T \mathbf{x}, \dots, \mathbf{x}^T \mathbf{x}]^T

, where

\mathbf{x} \in \mathbb{R}^{10}

. The term

\mathbf{x}^T \mathbf{x}

is the squared Euclidean norm of

\mathbf{x}

, denoted as

\|\mathbf{x}\|^2

. This is a scalar value. Thus, the mapping

\phi

produces a vector where each component is the same scalar value,

\|\mathbf{x}\|^2

. Let

k = \|\mathbf{x}\|^2

. Then

\phi(\mathbf{x}) = [k, k, \dots, k]^T

\mathbb{R}^{d}

. This means that the image of the mapping

\phi

is a subspace spanned by a single vector. For example, if

d=15

, then

\phi(\mathbf{x}) = [k, k, \dots, k]^T

which can be written as

k \cdot [1, 1, \dots, 1]^T

. The vector

[1, 1, \dots, 1]^T

forms a basis for the image space. Therefore, the dimension of the image space, which is equivalent to its rank, is 1, regardless of the value of

d

(as long as

d \ge 1

). The problem states that

d=15

, but the effective rank of the image of this specific feature map is always 1, as all output vectors lie on a single line passing through the origin.

📝 Chapter Quiz

Questions Level

Probability Distributions & Maximum Likelihood (MLE)

🤖 AI Math Unit 5 — Distributions & Estimation Applied STEM · Gaussian / Normal distribution N(μ, σ²) · Bernoulli & Binomial distributions for classification · Likelihood function L(θ | data) · Log-Likelihood log L(θ) and differentiation · MLE parameter derivation for Gaussian mean and variance

Data in AI is modeled using probability distributions. Maximum Likelihood Estimation (MLE) is the statistical method used to find the model parameters that make the observed training data most probable.

● Definition 5.1 — Maximum Likelihood Estimation (MLE)

Given independent and identically distributed (i.i.d.) observations

X = \{x_1, x_2, \dots, x_N\}

from a distribution parameterized by

\theta

:
1. Likelihood:

L(\theta) = \prod_{i=1}^N p(x_i | \theta)

.
2. Log-Likelihood:

\ell(\theta) = \ln L(\theta) = \sum_{i=1}^N \ln p(x_i | \theta)

. Taking the logarithm converts products into sums without shifting the maximizing argument.
3. MLE Estimator:

\hat{\theta}_{\text{MLE}} = \arg\max_\theta \ell(\theta)

, solved by setting

\frac{\partial \ell}{\partial \theta} = 0

● Deep Conceptual Intuition — Probability Distributions & Maximum Likelihood (MLE)

Coordinate intersection of linear system

y = 2x + 1

and

y = -x + 7

at unique equilibrium point

(2, 5)

⊙ Explorer · Maximum Likelihood Estimation (MLE): Gaussian Log-Likelihood ℓ

(\mu)

interactive

Live numerical table for Maximum Likelihood Estimation (MLE): Gaussian Log-Likelihood ℓ $(\mu)$
Parameter / Metric	Value

Candidate Mean $\mu =$ 0.5

Candidate Std Dev $\sigma =$ 1.2

Log-Likelihood ℓ

(\mu, \sigma)

ℓ = -10.36

True Sample Mean

\hat{\mu }_{\text{MLE}} = \bar{x}

\hat{\mu }_{\text{MLE}} = 1.50

Score Gradient ∂ℓ/∂

\mu

∂ℓ/∂\mu = 3.47

Given 5 observed data points on the horizontal axis, slide candidate mean $\mu$ to align the Gaussian bell (indigo) with the data and maximize the log-likelihood curve ℓ $(\mu)$ (amber).

Probability Distributions & Maximum Likelihood (MLE) — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Probability Distributions & Maximum Likelihood (MLE).

Gaussian PDF

p(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}

Gaussian Mean MLE

\hat{\mu}_{\text{MLE}} = \frac{1}{N}\sum_{i=1}^N x_i

(sample mean)

Gaussian Var MLE

\hat{\sigma}^2_{\text{MLE}} = \frac{1}{N}\sum_{i=1}^N (x_i - \hat{\mu})^2

Bernoulli MLE

\hat{p}_{\text{MLE}} = \frac{k}{N}

(fraction of positive trials)

◆ Why It Works — First-Principles Derivation (Probability Distributions & Maximum Likelihood (MLE))

Q_{\text{final}} = Q_0\left(1 + \frac{p}{100}\right)\!\left(1 + \frac{q}{100}\right) = Q_0\left(1 + \frac{p + q}{100} + \frac{pq}{10000}\right)

Worked Example 5.A — Deriving the MLE of a Bernoulli Parameter

Let $x_1, x_2, \dots, x_N \in \{0, 1\}$ be $N$ binary outcomes from a coin toss with unknown bias $p$ . Deriving the MLE estimator for $p$ .

Solution:
The probability of single outcome is $P(x_i) = p^{x_i}(1-p)^{1-x_i}$ .
The likelihood for $N$ i.i.d. observations is:
$L(p) = \prod_{i=1}^N p^{x_i}(1-p)^{1-x_i} = p^{\sum x_i} (1-p)^{N - \sum x_i}$
Let $k = \sum_{i=1}^N x_i$ be the total number of successes.
Log-likelihood:
$\ell(p) = \ln L(p) = k \ln p + (N - k) \ln(1 - p)$
Differentiate with respect to $p$ and set to 0:
$\frac{d\ell}{dp} = \frac{k}{p} - \frac{N - k}{1 - p} = 0 \implies \frac{k}{p} = \frac{N - k}{1 - p}$
$k(1 - p) = p(N - k) \implies k - kp = Np - kp \implies Np = k \implies \hat{p}_{\text{MLE}} = \frac{k}{N}$ .
Checking second derivative: $ \frac{d^2\ell}{dp^2} = -\frac{k}{p^2} - \frac{N-k}{(1-p)^2} < 0 $, confirming a maximum.
Answer: $\hat{p}_{\text{MLE}} = \frac{1}{N}\sum_{i=1}^N x_i$ .

Worked Example

B

— Probability Distributions & Maximum Likelihood (MLE) (Medium Application)

Problem (Medium): Principal Component Analysis on covariance matrix $\Sigma_{A}$ yields eigenvalues $\lambda_1 = 51$ and $\lambda_2 = 49$ . Determine the percentage of variance $f_{x}$ explained by the first principal component.

Step-by-Step Solution: The percentage explained is $\frac{51}{51+49} \times 100\% = 51\%$ .

Worked Example

C

— Probability Distributions & Maximum Likelihood (MLE) (Hard Application)

Problem (Hard): Consider an i.i.d. sample $x_1, x_2, \dots, x_n$ from a Poisson distribution with probability mass function $P(X = k; \theta) = \frac{\theta^k e^{-\theta}}{k!}$ for $k = 0, 1, 2, \dots$ , where $\theta > 0$ . Given a sample of size $n = 5$ with sample sum $\sum_{i=1}^5 x_i = 15$ , find the maximum likelihood estimate (MLE) $\hat{\theta}$ of $\theta$ .

Step-by-Step Solution: The log-likelihood function for $n$ i.i.d. Poisson(\/ $\theta$ ) observations is: $\ell(\theta) = \sum_{i=1}^n x_i \ln\theta - n\theta - \sum_{i=1}^n \ln(x_i!).$ Setting the derivative with respect to $\theta$ equal to zero gives: $\frac{d\ell}{d\theta} = \frac{\sum_{i=1}^n x_i}{\theta} - n = 0 \implies \hat{\theta} = \frac{1}{n}\sum_{i=1}^n x_i = \frac{15}{5} = 3.$

Worked Example

D

— Probability Distributions & Maximum Likelihood (MLE) (Easy Application)

Problem (Easy): Consider a discrete random variable $X$ with the following probability mass function (PMF): $P(X=x) = \frac{x}{10}$ for $x \in \{1, 2, 3, 4\}$ . What is the value of $P(X \le 2)$ ?

Step-by-Step Solution: The probability mass function is given by $P(X=x) = \frac{x}{10}$ for $x \in \{1, 2, 3, 4\}$ . We need to find $P(X \le 2)$ . This is the sum of probabilities for $X=1$ and $X=2$ . $P(X \le 2) = P(X=1) + P(X=2)$ . $P(X=1) = \frac{1}{10}$ and $P(X=2) = \frac{2}{10}$ . Therefore, $P(X \le 2) = \frac{1}{10} + \frac{2}{10} = \frac{3}{10}$ .

⚠️ Common Exam Traps — Probability Distributions & Maximum Likelihood (MLE)

2. Failing to Flip the Inequality Sign When Multiplying/Dividing by a Negative: In $-3x + 3 \le 6$ , dividing by $-3$ strictly reverses $\le$ to $\ge$ .

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Probability Distributions & Maximum Likelihood (MLE))

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Claiming $+25\%$ followed by $-25\%$ returns to $100\%$	The $25\%$ reduction applies to the larger base $1.25 Q_0$ , not $Q_0$ .	Multiply scale factors: $1.25 \times 0.75 = 0.9375$ (a $6.25\%$ net decrease).
Solving $-3x > 12$ as $x > -4$	Dividing by a negative number reverses order on the real number line.	Flip the inequality symbol whenever multiplying or dividing by a negative: $x < -4$.
Computing round-trip average speed as $\frac{v_1 + v_2}{2}$	Time spent at the slower speed is longer than time spent at the faster speed.	Use total distance over total time: $v_{\text{avg}} = \frac{2v_1 v_2}{v_1 + v_2}$ .

▲ Exam Pacing & Verification Strategy — Probability Distributions & Maximum Likelihood (MLE)

[Practice 1 · Medium] A dataset consists of observations

x_1, x_2, \dots, x_n

that are assumed to be drawn independently from a Bernoulli distribution with parameter

p

. The probability mass function is given by

P(X=k; p) = p^k (1-p)^{1-k}

for

k \in \{0, 1\}

. The following bar chart shows the frequency distribution of these observations, where

n = 50

{ "type": "bar", "labels": ["0", "1"], "data": [20, 30], "title": "Frequency Distribution of Bernoulli Observations" } Using the method of Maximum Likelihood Estimation (MLE), determine the estimated value of

p

for this distribution.

💡 Hint 1 of 3

The Bernoulli distribution has the probability mass function

P(X=k; p) = p^k (1-p)^{1-k}

. The likelihood function for

n

independent observations is

L(p; x_1, \dots, x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}

. This can be rewritten as

L(p) = p^{\sum x_i} (1-p)^{n - \sum x_i}

💡 Hint 2 of 3

Rewrite and simplify to obtain:

\sum x_i = np

💡 Hint 3 of 3

Therefore, the MLE for

p

is:

\hat{p}_{MLE} = \frac{30}{50}

The Bernoulli distribution has the probability mass function

P(X=k; p) = p^k (1-p)^{1-k}

. The likelihood function for

n

independent observations is

L(p; x_1, \dots, x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}

. This can be rewritten as

L(p) = p^{\sum x_i} (1-p)^{n - \sum x_i}

. The log-likelihood function is

\ln L(p) = (\sum x_i) \ln p + (n - \sum x_i) \ln(1-p)

. To find the maximum likelihood estimate (MLE) of

p

, we take the derivative of the log-likelihood with respect to

p

and set it to zero:

\frac{d}{dp} \ln L(p) = \frac{\sum x_i}{p} - \frac{n - \sum x_i}{1-p}

Setting the derivative to zero:

\frac{\sum x_i}{p} = \frac{n - \sum x_i}{1-p}

(\sum x_i)(1-p) = p(n - \sum x_i)

\sum x_i - p \sum x_i = np - p \sum x_i

\sum x_i = np

\hat{p}_{MLE} = \frac{\sum x_i}{n}

This means the MLE for

p

is the sample mean. From the bar chart, the frequency of observations equal to 0 is 20, and the frequency of observations equal to 1 is 30. The total number of observations is

n = 50

. The sum of the observations

\sum x_i

is calculated as:

\sum x_i = (20 \times 0) + (30 \times 1) = 0 + 30 = 30

Therefore, the MLE for

p

is:

\hat{p}_{MLE} = \frac{30}{50} = \frac{3}{5}

Common algebraic traps: Students might incorrectly calculate the sample mean, or confuse the counts of 0s and 1s. The option

\frac{2}{5}

would arise from incorrectly using

\frac{20}{50}

which represents

1-p

. The option

\frac{1}{2}

might arise from assuming equal probabilities without considering the data. The option

\frac{3}{2}

is not a valid probability.The Bernoulli distribution has the probability mass function

P(X=k; p) = p^k (1-p)^{1-k}

. The likelihood function for

n

independent observations is

L(p; x_1, \dots, x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i}

. This can be rewritten as

L(p) = p^{\sum x_i} (1-p)^{n - \sum x_i}

. The log-likelihood function is

\ln L(p) = (\sum x_i) \ln p + (n - \sum x_i) \ln(1-p)

. To find the maximum likelihood estimate (MLE) of

p

, we take the derivative of the log-likelihood with respect to

p

and set it to zero:

\frac{d}{dp} \ln L(p) = \frac{\sum x_i}{p} - \frac{n - \sum x_i}{1-p}

Setting the derivative to zero:

\frac{\sum x_i}{p} = \frac{n - \sum x_i}{1-p}

(\sum x_i)(1-p) = p(n - \sum x_i)

\sum x_i - p \sum x_i = np - p \sum x_i

\sum x_i = np

\hat{p}_{MLE} = \frac{\sum x_i}{n}

This means the MLE for

p

is the sample mean. From the bar chart, the frequency of observations equal to 0 is 20, and the frequency of observations equal to 1 is 30. The total number of observations is

n = 50

. The sum of the observations

\sum x_i

is calculated as:

\sum x_i = (20 \times 0) + (30 \times 1) = 0 + 30 = 30

Therefore, the MLE for

p

is:

\hat{p}_{MLE} = \frac{30}{50} = \frac{3}{5}

Common algebraic traps: Students might incorrectly calculate the sample mean, or confuse the counts of 0s and 1s. The option

\frac{2}{5}

would arise from incorrectly using

\frac{20}{50}

which represents

1-p

. The option

\frac{1}{2}

might arise from assuming equal probabilities without considering the data. The option

\frac{3}{2}

is not a valid probability.

[Practice 2 · Hard] Consider a sequence of

n

independent Bernoulli trials, where the probability of success in each trial is

p

. Let

X

be the random variable representing the number of successes in these

n

trials. Thus,

X \sim \text{Binomial}(n, p)

. We are given a dataset of

N

observed values

\{x_1, x_2, \dots, x_N\}

for

X

, where each

x_i

is an observed count of successes from

n

trials. The observed data are: [3, 4, 2, 5, 4, 3, 2, 4, 5, 3]. If we assume that

n=5

(the number of trials for each observation is fixed at 5), find the value of

p

that maximizes the likelihood of observing this dataset. Then, compute the maximum likelihood estimate (MLE) of the probability of success,

p

, and round your answer to two decimal places.

💡 Hint 1 of 3

The random variable

X

follows a Binomial distribution:

X \sim \text{Binomial}(n, p)

. The probability mass function (PMF) of the Binomial distribution is given by

P(X=k | n, p) = \binom{n}{k} p^k (1-p)^{n-k}

, where

k

is the number of successes.

💡 Hint 2 of 3

Rewrite and simplify to obtain:

\frac{S}{p} = \frac{T}{1-p}

💡 Hint 3 of 3

Let's check the second derivative to confirm it's a maximum:

\frac{d^2\ell}{dp^2} = -\frac{S}{p^2} - \frac{T}{(1-p)^2}

. Since

S>0

T>0

p \in (0,1)

, the second derivative is negative, confirming a maximum.

The random variable

X

follows a Binomial distribution:

X \sim \text{Binomial}(n, p)

. The probability mass function (PMF) of the Binomial distribution is given by

P(X=k | n, p) = \binom{n}{k} p^k (1-p)^{n-k}

, where

k

is the number of successes. We are given

n=5

and a dataset of

N=10

observations:

\{x_1, x_2, \dots, x_{10}\} = \{3, 4, 2, 5, 4, 3, 2, 4, 5, 3\}

. The likelihood function for this dataset, given

p

, is the product of the PMFs for each observation:

L(p | \{x_i\}) = \prod_{i=1}^{N} P(X=x_i | n, p) = \prod_{i=1}^{10} \binom{5}{x_i} p^{x_i} (1-p)^{5-x_i}

To find the MLE of

p

, we typically work with the log-likelihood function, which simplifies the product into a sum:

\ell(p) = \log L(p) = \sum_{i=1}^{10} \log \left( \binom{5}{x_i} p^{x_i} (1-p)^{5-x_i} \right)

\ell(p) = \sum_{i=1}^{10} \left( \log \binom{5}{x_i} + x_i \log p + (5-x_i) \log (1-p) \right)

\ell(p) = \sum_{i=1}^{10} \log \binom{5}{x_i} + \left(\sum_{i=1}^{10} x_i\right) \log p + \left(\sum_{i=1}^{10} (5-x_i)\right) \log (1-p)

The term

\sum_{i=1}^{10} \log \binom{5}{x_i}

is a constant with respect to

p

and can be ignored for maximization. Let

S = \sum_{i=1}^{10} x_i

be the total number of successes in the dataset, and

T = \sum_{i=1}^{10} (5-x_i) = 10 \times 5 - \sum_{i=1}^{10} x_i = 50 - S

be the total number of failures. So,

\ell(p) \approx S \log p + T \log (1-p)

. To find the MLE, we take the derivative of

\ell(p)

with respect to

p

and set it to zero:

\frac{d\ell}{dp} = \frac{S}{p} - \frac{T}{1-p} = 0

\frac{S}{p} = \frac{T}{1-p}

S(1-p) = Tp

S - Sp = Tp

S = Tp + Sp

S = p(T+S)

p = \frac{S}{T+S}

We know that

T+S = \sum_{i=1}^{10} (5-x_i) + \sum_{i=1}^{10} x_i = \sum_{i=1}^{10} (5-x_i+x_i) = \sum_{i=1}^{10} 5 = 10 \times 5 = 50

. Now, let's calculate

S

, the sum of the observed successes:

S = 3 + 4 + 2 + 5 + 4 + 3 + 2 + 4 + 5 + 3 = 35

. Therefore, the MLE of

p

is:

\hat{p} = \frac{S}{nN} = \frac{35}{5 \times 10} = \frac{35}{50} = \frac{7}{10} = 0.70

. The maximum likelihood estimate of

p

is 0.70. Let's check the second derivative to confirm it's a maximum:

\frac{d^2\ell}{dp^2} = -\frac{S}{p^2} - \frac{T}{(1-p)^2}

. Since

S>0

T>0

p \in (0,1)

, the second derivative is negative, confirming a maximum. Common Misconception: A common mistake might be to incorrectly calculate the sum of successes or failures, or to use the wrong formula for the MLE of a binomial proportion, such as simply averaging the proportions of successes in each trial (which is not applicable here as

n

is fixed). For example, if one made an arithmetic error and got

S=30

, then

\hat{p} = 30/50 = 0.60

. If

S=37.5

(an impossible sum, but for illustration of error),

\hat{p}=37.5/50 = 0.75

. If

S=30

\hat{p}=30/50=0.6

. If

S=33.5

\hat{p}=33.5/50=0.67

.The random variable

X

follows a Binomial distribution:

X \sim \text{Binomial}(n, p)

. The probability mass function (PMF) of the Binomial distribution is given by

P(X=k | n, p) = \binom{n}{k} p^k (1-p)^{n-k}

, where

k

is the number of successes. We are given

n=5

and a dataset of

N=10

observations:

\{x_1, x_2, \dots, x_{10}\} = \{3, 4, 2, 5, 4, 3, 2, 4, 5, 3\}

. The likelihood function for this dataset, given

p

, is the product of the PMFs for each observation:

L(p | \{x_i\}) = \prod_{i=1}^{N} P(X=x_i | n, p) = \prod_{i=1}^{10} \binom{5}{x_i} p^{x_i} (1-p)^{5-x_i}

To find the MLE of

p

, we typically work with the log-likelihood function, which simplifies the product into a sum:

\ell(p) = \log L(p) = \sum_{i=1}^{10} \log \left( \binom{5}{x_i} p^{x_i} (1-p)^{5-x_i} \right)

\ell(p) = \sum_{i=1}^{10} \left( \log \binom{5}{x_i} + x_i \log p + (5-x_i) \log (1-p) \right)

\ell(p) = \sum_{i=1}^{10} \log \binom{5}{x_i} + \left(\sum_{i=1}^{10} x_i\right) \log p + \left(\sum_{i=1}^{10} (5-x_i)\right) \log (1-p)

The term

\sum_{i=1}^{10} \log \binom{5}{x_i}

is a constant with respect to

p

and can be ignored for maximization. Let

S = \sum_{i=1}^{10} x_i

be the total number of successes in the dataset, and

T = \sum_{i=1}^{10} (5-x_i) = 10 \times 5 - \sum_{i=1}^{10} x_i = 50 - S

be the total number of failures. So,

\ell(p) \approx S \log p + T \log (1-p)

. To find the MLE, we take the derivative of

\ell(p)

with respect to

p

and set it to zero:

\frac{d\ell}{dp} = \frac{S}{p} - \frac{T}{1-p} = 0

\frac{S}{p} = \frac{T}{1-p}

S(1-p) = Tp

S - Sp = Tp

S = Tp + Sp

S = p(T+S)

p = \frac{S}{T+S}

We know that

T+S = \sum_{i=1}^{10} (5-x_i) + \sum_{i=1}^{10} x_i = \sum_{i=1}^{10} (5-x_i+x_i) = \sum_{i=1}^{10} 5 = 10 \times 5 = 50

. Now, let's calculate

S

, the sum of the observed successes:

S = 3 + 4 + 2 + 5 + 4 + 3 + 2 + 4 + 5 + 3 = 35

. Therefore, the MLE of

p

is:

\hat{p} = \frac{S}{nN} = \frac{35}{5 \times 10} = \frac{35}{50} = \frac{7}{10} = 0.70

. The maximum likelihood estimate of

p

is 0.70. Let's check the second derivative to confirm it's a maximum:

\frac{d^2\ell}{dp^2} = -\frac{S}{p^2} - \frac{T}{(1-p)^2}

. Since

S>0

T>0

p \in (0,1)

n

is fixed). For example, if one made an arithmetic error and got

S=30

, then

\hat{p} = 30/50 = 0.60

. If

S=37.5

(an impossible sum, but for illustration of error),

\hat{p}=37.5/50 = 0.75

. If

S=30

\hat{p}=30/50=0.6

. If

S=33.5

\hat{p}=33.5/50=0.67

[Practice 3 · Easy] Consider a dataset where the number of defective items produced by a machine per hour,

Y

, follows a Poisson distribution with a rate parameter

\lambda = 3

. The probability mass function (PMF) for a Poisson distribution is given by

P(Y=k) = \frac{\lambda^k e^{-\lambda}}{k!}

for

k = 0, 1, 2, \dots

. What is the value of

\lambda

in this scenario?

💡 Hint 1 of 3

The problem statement explicitly defines that the number of defective items,

Y

, follows a Poisson distribution with a rate parameter

\lambda = 3

💡 Hint 2 of 3

The rate parameter

\lambda

is a direct characteristic of the Poisson distribution that determines its mean and variance.

💡 Hint 3 of 3

Therefore, the value of

\lambda

is directly given as 3.

The problem statement explicitly defines that the number of defective items,

Y

, follows a Poisson distribution with a rate parameter

\lambda = 3

. The rate parameter

\lambda

is a direct characteristic of the Poisson distribution that determines its mean and variance. Therefore, the value of

\lambda

is directly given as 3.

[Practice 4 · Easy] A feature in an AI vision model follows a normal distribution

X \sim \mathcal{N}(25, 36)

. What is the mean

\mu

and variance

\sigma^2

of this feature?

💡 Hint 1 of 3

Start by examining the given mathematical expression in the problem:

X \sim \mathcal{N}(25, 36)

💡 Hint 2 of 3

The notation

\mathcal{N}(\mu, \sigma^2)

directly specifies the mean

\mu = 25

and variance

\sigma^2 = 36

(with standard deviation

\sigma

💡 Hint 3 of 3

Apply this step to

\sigma

and simplify to determine the final result.

The notation

\mathcal{N}(\mu, \sigma^2)

directly specifies the mean

\mu = 25

and variance

\sigma^2 = 36

(with standard deviation

\sigma = 6

[Practice 5 · Medium] A dataset consists of three independent observations

X_1, X_2, X_3

drawn from a Poisson distribution with parameter

\lambda

. The observed values are

x_1 = 3, x_2 = 5, x_3 = 4

. We want to find the Maximum Likelihood Estimate (MLE) for

\lambda

. The log-likelihood function for a Poisson distribution is given by

\ell(\lambda) = \sum_{i=1}^{n} \left( x_i \log(\lambda) - \log(x_i!) - \lambda \right)

. Part 1: Derive the expression for the derivative of the log-likelihood function with respect to

\lambda

. Part 2: Set the derivative to zero to find the MLE of

\lambda

💡 Hint 1 of 3

The log-likelihood function for

n=3

observations is

\ell(\lambda) = (x_1 \log(\lambda) - \log(x_1!) - \lambda) + (x_2 \log(\lambda) - \log(x_2!) - \lambda) + (x_3 \log(\lambda) - \log(x_3!) - \lambda)

💡 Hint 2 of 3

Rewrite and simplify to obtain:

\frac{x_1 + x_2 + x_3}{\lambda} - 3 = 0

💡 Hint 3 of 3

Substituting the observed values

x_1 = 3, x_2 = 5, x_3

\hat{\lambda} = \frac{3+5+4}{3}

The log-likelihood function for

n=3

observations is

\ell(\lambda) = (x_1 \log(\lambda) - \log(x_1!) - \lambda) + (x_2 \log(\lambda) - \log(x_2!) - \lambda) + (x_3 \log(\lambda) - \log(x_3!) - \lambda)

. Part 1: The derivative of the log-likelihood function with respect to

\lambda

is:

\frac{d\ell}{d\lambda} = \sum_{i=1}^{3} \left( \frac{x_i}{\lambda} - 1 \right) = \left( \frac{x_1}{\lambda} - 1 \right) + \left( \frac{x_2}{\lambda} - 1 \right) + \left( \frac{x_3}{\lambda} - 1 \right)

. Part 2: To find the MLE, we set the derivative to zero:

\frac{x_1}{\lambda} - 1 + \frac{x_2}{\lambda} - 1 + \frac{x_3}{\lambda} - 1 = 0

\frac{x_1 + x_2 + x_3}{\lambda} - 3 = 0

\frac{x_1 + x_2 + x_3}{\lambda} = 3

\hat{\lambda} = \frac{x_1 + x_2 + x_3}{3}

. Substituting the observed values

x_1 = 3, x_2 = 5, x_3 = 4

\hat{\lambda} = \frac{3+5+4}{3} = \frac{12}{3} = 4

. Common algebraic traps include incorrectly differentiating the log-likelihood or making errors in solving for

\lambda

. For example, forgetting the

-1

term from the derivative of

-\lambda

leads to an incorrect denominator. Choice B is incorrect as it includes

e

which is not part of the Poisson MLE derivation. Choice C is an incomplete and incorrect expression. Choice D is only correct if the sum of observations was 12, but the formula for

\hat{\lambda}

is the sample mean. Therefore, the MLE is

\hat{\lambda} = 4

.The log-likelihood function for

n=3

observations is

\ell(\lambda) = (x_1 \log(\lambda) - \log(x_1!) - \lambda) + (x_2 \log(\lambda) - \log(x_2!) - \lambda) + (x_3 \log(\lambda) - \log(x_3!) - \lambda)

. Part 1: The derivative of the log-likelihood function with respect to

\lambda

is:

\frac{d\ell}{d\lambda} = \sum_{i=1}^{3} \left( \frac{x_i}{\lambda} - 1 \right) = \left( \frac{x_1}{\lambda} - 1 \right) + \left( \frac{x_2}{\lambda} - 1 \right) + \left( \frac{x_3}{\lambda} - 1 \right)

. Part 2: To find the MLE, we set the derivative to zero:

\frac{x_1}{\lambda} - 1 + \frac{x_2}{\lambda} - 1 + \frac{x_3}{\lambda} - 1 = 0

\frac{x_1 + x_2 + x_3}{\lambda} - 3 = 0

\frac{x_1 + x_2 + x_3}{\lambda} = 3

\hat{\lambda} = \frac{x_1 + x_2 + x_3}{3}

. Substituting the observed values

x_1 = 3, x_2 = 5, x_3 = 4

\hat{\lambda} = \frac{3+5+4}{3} = \frac{12}{3} = 4

. Common algebraic traps include incorrectly differentiating the log-likelihood or making errors in solving for

\lambda

. For example, forgetting the

-1

term from the derivative of

-\lambda

leads to an incorrect denominator. Choice B is incorrect as it includes

e

which is not part of the Poisson MLE derivation. Choice C is an incomplete and incorrect expression. Choice D is only correct if the sum of observations was 12, but the formula for

\hat{\lambda}

is the sample mean. Therefore, the MLE is

\hat{\lambda} = 4

📝 Chapter Quiz

Questions Level

Linear Regression & Ordinary Least Squares

🤖 AI Math Unit 6 — Linear Models & OLS Applied STEM · Simple linear regression: y = mx + b · Matrix formulation Y = Xw + ε · Ordinary Least Squares (OLS) closed form · Normal equation w = (XᵀX)⁻¹ Xᵀ y · Residuals, MSE and R² coefficient of determination

Linear regression is the foundational predictive model in data science. We explore both algebraic least-squares minimization and the closed-form matrix Normal Equation used across analytics libraries.

▲ Theorem 6.1 — The Matrix Normal Equation

For a linear model

\mathbf{y} = \mathbf{X}\mathbf{w}

where

\mathbf{X} \in \mathbb{R}^{N \times (d+1)}

is the design matrix (including a column of ones for intercept), the weight vector

\mathbf{w}

that minimizes the sum of squared errors

\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2

satisfies:

\mathbf{X}^T \mathbf{X} \mathbf{w} = \mathbf{X}^T \mathbf{y} \implies \mathbf{w} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}

provided

\mathbf{X}^T \mathbf{X}

is invertible (non-singular).

● Deep Conceptual Intuition — Linear Regression & Ordinary Least Squares

Coordinate intersection of linear system

y = 2x + 1

and

y = -x + 7

at unique equilibrium point

(2, 5)

⊙ Explorer · Ordinary Least Squares (OLS) Regression Line & Residual MSE

interactive

Live numerical table for Ordinary Least Squares (OLS) Regression Line & Residual MSE
Parameter / Metric	Value

Candidate Weight (Slope) $w =$ 0.8

Candidate Bias (Intercept) $b =$ 0.5

Mean Squared Error MSE

(w, b)

MSE = 0.124

Optimal Normal-Eq

(w*, b*)

w* = 1.04, b* = 0.50

Coefficient of Determination

R^{2}

R^{2} = 0.943

Vertical amber segments show the residuals $e_i = y_i -(w \cdot x_i + b)$ for each training point. Minimizing $\Sigma$ e_ $i^{2}$ yields the dashed ordinary least squares line.

Regression Line and Observations — illustrating key geometric and analytical relationships in Linear Regression & Ordinary Least Squares.

Slope Formula

m = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}}

Intercept

b = \bar{y} - m\bar{x}

R² Score

R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} \in [0, 1]

Residual Vector

\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}} \perp \text{col}(\mathbf{X})

◆ Why It Works — First-Principles Derivation (Linear Regression & Ordinary Least Squares)

Q_{\text{final}} = Q_0\left(1 + \frac{p}{100}\right)\!\left(1 + \frac{q}{100}\right) = Q_0\left(1 + \frac{p + q}{100} + \frac{pq}{10000}\right)

Worked Example 6.A — Closed-Form Matrix Normal Equation

Fit a line $y = w_1 x + w_0$ to three points: $(1, 2)$ , $(2, 3)$ , and $(3, 5)$ using the Normal Equation.

Solution:
1. Set up design matrix $\mathbf{X}$ (with bias column) and target $\mathbf{y}$ :
$\mathbf{X} = \begin{pmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{pmatrix}, \quad \mathbf{y} = \begin{pmatrix} 2 \\ 3 \\ 5 \end{pmatrix}$
2. Compute $\mathbf{X}^T \mathbf{X}$ :
$\mathbf{X}^T \mathbf{X} = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{pmatrix} = \begin{pmatrix} 1+4+9 & 1+2+3 \\ 1+2+3 & 1+1+1 \end{pmatrix} = \begin{pmatrix} 14 & 6 \\ 6 & 3 \end{pmatrix}$
3. Invert $\mathbf{X}^T \mathbf{X}$ :
$\det = (14)(3) - (6)(6) = 42 - 36 = 6$ .
$(\mathbf{X}^T \mathbf{X})^{-1} = \frac{1}{6} \begin{pmatrix} 3 & -6 \\ -6 & 14 \end{pmatrix}$
4. Compute $\mathbf{X}^T \mathbf{y}$ :
$\mathbf{X}^T \mathbf{y} = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 3 \\ 5 \end{pmatrix} = \begin{pmatrix} 2 + 6 + 15 \\ 2 + 3 + 5 \end{pmatrix} = \begin{pmatrix} 23 \\ 10 \end{pmatrix}$
5. Compute $\mathbf{w} = (\mathbf{X}^T \mathbf{X})^{-1} (\mathbf{X}^T \mathbf{y})$ :
$\mathbf{w} = \frac{1}{6} \begin{pmatrix} 3(23) - 6(10) \\ -6(23) + 14(10) \end{pmatrix} = \frac{1}{6} \begin{pmatrix} 69 - 60 \\ -138 + 140 \end{pmatrix} = \frac{1}{6} \begin{pmatrix} 9 \\ 2 \end{pmatrix} = \begin{pmatrix} 1.5 \\ 0.333 \end{pmatrix}$
Line equation: $\hat{y} = 1.5x + \frac{1}{3}$ .
Answer: Slope $w_1 = 1.5$ , Intercept $w_0 = \frac{1}{3}$ .

Fit a line $y = w_1 x + w_0$ to three points: $(1, 2)$ , $(2, 3)$ , and $(3, 5)$ using the Normal المعادلة.

Solution:
1. Set up design المصفوفة $\mathbf{X}$ (with bias column) and target $\mathbf{y}$ :
$\mathbf{X} = \begin{pmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{pmatrix}, \quad \mathbf{y} = \begin{pmatrix} 2 \\ 3 \\ 5 \end{pmatrix}$
2. Compute $\mathbf{X}^T \mathbf{X}$ :
$\mathbf{X}^T \mathbf{X} = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{pmatrix} = \begin{pmatrix} 1+4+9 & 1+2+3 \\ 1+2+3 & 1+1+1 \end{pmatrix} = \begin{pmatrix} 14 & 6 \\ 6 & 3 \end{pmatrix}$
3. Invert $\mathbf{X}^T \mathbf{X}$ :
$\det = (14)(3) - (6)(6) = 42 - 36 = 6$ .
$(\mathbf{X}^T \mathbf{X})^{-1} = \frac{1}{6} \begin{pmatrix} 3 & -6 \\ -6 & 14 \end{pmatrix}$
4. Compute $\mathbf{X}^T \mathbf{y}$ :
$\mathbf{X}^T \mathbf{y} = \begin{pmatrix} 1 & 2 & 3 \\ 1 & 1 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 3 \\ 5 \end{pmatrix} = \begin{pmatrix} 2 + 6 + 15 \\ 2 + 3 + 5 \end{pmatrix} = \begin{pmatrix} 23 \\ 10 \end{pmatrix}$
5. Compute $\mathbf{w} = (\mathbf{X}^T \mathbf{X})^{-1} (\mathbf{X}^T \mathbf{y})$ :
$\mathbf{w} = \frac{1}{6} \begin{pmatrix} 3(23) - 6(10) \\ -6(23) + 14(10) \end{pmatrix} = \frac{1}{6} \begin{pmatrix} 69 - 60 \\ -138 + 140 \end{pmatrix} = \frac{1}{6} \begin{pmatrix} 9 \\ 2 \end{pmatrix} = \begin{pmatrix} 1.5 \\ 0.333 \end{pmatrix}$
Line المعادلة: $\hat{y} = 1.5x + \frac{1}{3}$ .
Answer: الميل $w_1 = 1.5$ , المقطع $w_0 = \frac{1}{3}$ .

Worked Example

B

— Linear Regression & Ordinary Least Squares (Hard Application)

Problem (Hard): A simple linear regression model has been fitted to a dataset. The model equation is given by $\hat{y} = ax + b$ . The regression line passes through the points $(2, 10)$ and $(5, 22)$ in the feature-target space. Determine the value of the intercept $b$ .

Step-by-Step Solution: The problem states that the linear regression line $\hat{y} = ax + b$ passes through two points: $(x_1, y_1) = (2, 10)$ and $(x_2, y_2) = (5, 22)$ . We can use these two points to form a system of linear equations to solve for the unknown parameters $a$ (the slope) and $b$ (the intercept). Substituting the first point $(2, 10)$ into the equation $\hat{y} = ax + b$ gives: $10 = a(2) + b$ (Equation 1) Substituting the second point $(5, 22)$ into the equation $\hat{y} = ax + b$ gives: $22 = a(5) + b$ (Equation 2) To find the slope $a$ , we can subtract Equation 1 from Equation 2: $(22 - 10) = (5a - 2a) + (b - b)$ $12 = 3a$ $a = \frac{12}{3} = 4$ Now that we have the value of $a$ , we can substitute it back into either Equation 1 or Equation 2 to solve for $b$ . Let's use Equation 1: $10 = 4(2) + b$ $10 = 8 + b$ $b = 10 - 8$ $b = 2$ Let's verify with Equation 2: $22 = 4(5) + b$ $22 = 20 + b$ $b = 22 - 20$ $b = 2$ Both equations yield $b = 2$ . Therefore, the intercept $b$ is 2. Common algebraic traps include incorrectly calculating the slope or making sign errors when substituting values back into the equations. For instance, if one incorrectly calculated the slope as $\frac{3}{12}$ or made a sign error in $10 - 8$ , they might arrive at an incorrect intercept.

مسألة (Hard): A simple linear regression model has been fitted to a dataset. The model equation is given by $\hat{y} = ax + b$ . The regression line passes through the points $(2, 10)$ and $(5, 22)$ in the feature-target space. Determine the value of the intercept $b$ .

الحل خطوة بخطوة: The problem states that the linear regression line $\hat{y} = ax + b$ passes through two points: $(x_1, y_1) = (2, 10)$ and $(x_2, y_2) = (5, 22)$ . We can use these two points to form a system of linear equations to solve for the unknown parameters $a$ (the slope) and $b$ (the intercept). Substituting the first point $(2, 10)$ into the equation $\hat{y} = ax + b$ gives: $10 = a(2) + b$ (Equation 1) Substituting the second point $(5, 22)$ into the equation $\hat{y} = ax + b$ gives: $22 = a(5) + b$ (Equation 2) To find the slope $a$ , we can subtract Equation 1 from Equation 2: $(22 - 10) = (5a - 2a) + (b - b)$ $12 = 3a$ $a = \frac{12}{3} = 4$ Now that we have the value of $a$ , we can substitute it back into either Equation 1 or Equation 2 to solve for $b$ . Let's use Equation 1: $10 = 4(2) + b$ $10 = 8 + b$ $b = 10 - 8$ $b = 2$ Let's verify with Equation 2: $22 = 4(5) + b$ $22 = 20 + b$ $b = 22 - 20$ $b = 2$ Both equations yield $b = 2$ . Therefore, the intercept $b$ is 2. Common algebraic traps include incorrectly calculating the slope or making sign errors when substituting values back into the equations. For instance, if one incorrectly calculated the slope as $\frac{3}{12}$ or made a sign error in $10 - 8$ , they might arrive at an incorrect intercept.

Worked Example

C

— Linear Regression & Ordinary Least Squares (Easy Application)

Problem (Easy): In gradient descent on loss $f(x)$ , parameter $x_0 = 30$ is updated with learning rate $\eta = 0.1$ and gradient $\nabla f(x_0) = 20$ . Determine the updated parameter $x_1 = x_0 - \eta \nabla f(x_0)$ .

Step-by-Step Solution: Computing $x_1 = 30 - 0.1(20) = 28$ .

Worked Example

D

— Linear Regression & Ordinary Least Squares (Medium Application)

Problem (Medium): Consider a simple linear regression model where the predicted value $\hat{y}$ is related to the independent variable $x$ by the equation $\hat{y} = mx + c$ . If the line of best fit passes through the points $(1, 5)$ and $(3, 11)$ , what is the slope $m$ of this line?

Step-by-Step Solution: The slope $m$ of a line passing through two points $(x_1, y_1)$ and $(x_2, y_2)$ is given by the formula $m = \frac{y_2 - y_1}{x_2 - x_1}$ . Given the points $(1, 5)$ and $(3, 11)$ , we have $x_1 = 1$ , $y_1 = 5$ , $x_2 = 3$ , and $y_2 = 11$ . Plugging these values into the formula, we get $m = \frac{11 - 5}{3 - 1} = \frac{6}{2} = 3$ . Therefore, the slope of the line of best fit is 3.

⚠️ Common Exam Traps — Linear Regression & Ordinary Least Squares

2. Failing to Flip the Inequality Sign When Multiplying/Dividing by a Negative: In $-2x + 5 \le 7$ , dividing by $-2$ strictly reverses $\le$ to $\ge$ .

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Linear Regression & Ordinary Least Squares)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Claiming $+25\%$ followed by $-25\%$ returns to $100\%$	The $25\%$ reduction applies to the larger base $1.25 Q_0$ , not $Q_0$ .	Multiply scale factors: $1.25 \times 0.75 = 0.9375$ (a $6.25\%$ net decrease).
Solving $-3x > 12$ as $x > -4$	Dividing by a negative number reverses order on the real number line.	Flip the inequality symbol whenever multiplying or dividing by a negative: $x < -4$.
Computing round-trip average speed as $\frac{v_1 + v_2}{2}$	Time spent at the slower speed is longer than time spent at the faster speed.	Use total distance over total time: $v_{\text{avg}} = \frac{2v_1 v_2}{v_1 + v_2}$ .

▲ Exam Pacing & Verification Strategy — Linear Regression & Ordinary Least Squares

[Practice 1 · Hard] A 2D loss surface is defined as

L(w_1, w_2) = 4w_1^2 + 7w_2^2

. Find the gradient vector

\nabla L

at the point

(w_1, w_2) = (2, -1)

💡 Hint 1 of 3

Compute partial derivatives:

\frac{\partial L}{\partial w_1} = 2(4)w_1 = 8(2)

💡 Hint 2 of 3

Rewrite and simplify to obtain:

\frac{\partial L}{\partial w_2} = 2(7)w_2 = 14(-1)

💡 Hint 3 of 3

Therefore,

\nabla L = [16, -14]^T

Compute partial derivatives:

\frac{\partial L}{\partial w_1} = 2(4)w_1 = 8(2) = 16

;

\frac{\partial L}{\partial w_2} = 2(7)w_2 = 14(-1) = -14

. Therefore,

\nabla L = [16, -14]^T

[Practice 2 · Easy] A fully-connected dense layer maps an input

\mathbf{x} \in \mathbb{R}^{50}

\mathbf{y} \in \mathbb{R}^{25}

using

\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}

. How many total learnable parameters are in this layer?

💡 Hint 1 of 3

The weight matrix

\mathbf{W}

has shape

25 \times 50

, giving

1250

weights.

💡 Hint 2 of 3

The bias vector

\mathbf{b}

has

25

parameters.

💡 Hint 3 of 3

Substitute the result from the previous step into the governing relation and verify the problem condition to find the final answer.

The weight matrix

\mathbf{W}

has shape

25 \times 50

, giving

1250

weights. The bias vector

\mathbf{b}

has

25

parameters. Total =

1250 + 25 = 1275

[Practice 3 · Easy] In a dataset, the prior probability of class

C_1

P(C_1) = 0.10

. What is the probability of the complement class

P(\neg C_1)

💡 Hint 1 of 3

By the complement rule of probability:

P( eg C_1) = 1 - P(C_1)

💡 Hint 2 of 3

Substitute and simplify:

P( eg C_1) = 1 - 0.10

💡 Hint 3 of 3

Compute the final value from:

1 - 0.10

By the complement rule of probability:

P( eg C_1) = 1 - P(C_1) = 1 - 0.10 = 0.90

[Practice 4 · Medium] Consider a simple linear regression model where a dependent variable

y

is predicted from an independent variable

x

. The model is given by the equation

y = \beta_0 + \beta_1 x + \epsilon

, where

\epsilon

is the error term. Suppose we have estimated the regression line to be

\hat{y} = 3 + 2x

. If we observe an independent variable value of

x=4

, what is the predicted value

\hat{y}

for the dependent variable?

💡 Hint 1 of 3

The problem asks for the predicted value

\hat{y}

given an independent variable value

x=4

and a linear regression equation

\hat{y} = 3 + 2x

. We substitute the value of

x

into the equation:

\hat{y} = 3 + 2(4)

💡 Hint 2 of 3

Rewrite and simplify to obtain:

\hat{y} = 3 + 8

\hat{y} = 11

💡 Hint 3 of 3

Therefore, the predicted value for the dependent variable is 11.

The problem asks for the predicted value

\hat{y}

given an independent variable value

x=4

and a linear regression equation

\hat{y} = 3 + 2x

. We substitute the value of

x

into the equation:

\hat{y} = 3 + 2(4)

\hat{y} = 3 + 8

\hat{y} = 11

Therefore, the predicted value for the dependent variable is 11. Common errors might involve incorrectly multiplying 2 by 4 or incorrectly adding 3 to 8.

[Practice 5 · Hard] In anomaly detection, anomaly prior

P(A) = 0.10

, detection sensitivity

P(+|A) = 0.95

, and false alarm rate

P(+|\neg A) = 0.05

. By Bayes' theorem, what is the posterior probability

P(A|+)

that an alarm is a true anomaly?

💡 Hint 1 of 3

By Bayes' Rule:

P(A|+) = \frac{P(+|A)P(A)}{P(+|A)P(A) + P(+| eg A)P( eg A)}

💡 Hint 2 of 3

Substitute and simplify:

P(A|+) = \frac{0.95 \times 0.10}{0.95(0.10) + 0.05(0.90)}

💡 Hint 3 of 3

Compute the final value from:

\frac{0.0950}{0.1400}

By Bayes' Rule:

P(A|+) = \frac{P(+|A)P(A)}{P(+|A)P(A) + P(+| eg A)P( eg A)} = \frac{0.95 \times 0.10}{0.95(0.10) + 0.05(0.90)} = \frac{0.0950}{0.1400} = 0.679

📝 Chapter Quiz

Questions Level

Logistic Regression & Classification Boundaries

🤖 AI Math Unit 7 — Logistic Classification Applied STEM · Binary classification & probabilities · The Sigmoid function σ(z) = 1 / (1 + e⁻ᶻ) · Log-odds (Logit) transformation · Binary Cross-Entropy (Log Loss) function · Linear decision boundaries and threshold tuning

When the target variable is categorical (0 or 1, positive or negative), standard linear regression fails. Logistic regression maps continuous linear combinations to calibrated probabilities using the Sigmoid curve.

● Definition 7.1 — Sigmoid & Binary Cross-Entropy

1. Sigmoid Function:

\sigma(z) = \frac{1}{1 + e^{-z}}

.
Key properties:

\sigma(0) = 0.5

\lim_{z \to \infty} \sigma(z) = 1

\lim_{z \to -\infty} \sigma(z) = 0

. Derivative:

\sigma'(z) = \sigma(z)(1 - \sigma(z))

.

2. Binary Cross-Entropy Loss: For true label

y \in \{0, 1\}

and model prediction

\hat{y} = \sigma(\mathbf{w}^T\mathbf{x})

L(y, \hat{y}) = -\Big[ y \ln\hat{y} + (1 - y) \ln(1 - \hat{y}) \Big]

● Deep Conceptual Intuition — Logistic Regression & Classification Boundaries

Accumulation, Antiderivatives & Differential Flows: By the Fundamental Theorem of Calculus, the accumulation function $A(x) = \int_a^x f(t)\,dt$ satisfies $A'(x) = f(x)$ , linking instantaneous rates of change to net signed area. In first-order differential equations $\dfrac{dy}{dx} = g(x)h(y)$ , separation of variables $\int \dfrac{1}{h(y)}\,dy = \int g(x)\,dx + C$ reconstructs the family of integral curves across the slope field.

Slope field for

\frac{dy}{dx} = x -y

with the integral curve passing through initial condition

(0, 1)

⊙ Explorer · Logistic Regression Sigmoid

\sigma

(wx +

b

) & Decision Boundary

interactive

Live numerical table for Logistic Regression Sigmoid $\sigma$ (wx + $b$ ) & Decision Boundary
Parameter / Metric	Value

Weight Steepness $w =$ 1.8

Bias Shift $b =$ -0.9

Decision Boundary x* =

-b / w

x* = 0.50

Probability

\sigma (w \cdot 1 + b)

\sigma (z) = 0.711

Binary Cross-Entropy LossBCE ≈ 0.389

Class 0 (indigo dots at $y = 0$ ) and Class 1 (amber dots at $y = 1$ ) are separated by the sigmoid curve $P(y=1|x) = 1$ /(1 + e^(−(wx+ $b$ ))), which crosses 0.5 at x* = $-b / w$ .

Binary Classification Contingency Table
	Pred Positive	Pred Negative
Actual Positive	45	10
Actual Negative	10	40

Binary Classification Contingency Table — illustrating key geometric and analytical relationships in Logistic Regression & Classification Boundaries.

Odds Ratio

\text{Odds} = \frac{p}{1-p} = e^{\mathbf{w}^T\mathbf{x}}

Log-Odds (Logit)

\ln\left(\frac{p}{1-p}\right) = \mathbf{w}^T\mathbf{x}

Decision Boundary

\hat{y} \ge 0.5 \iff \mathbf{w}^T\mathbf{x} \ge 0

(hyperplane separating classes)

Loss Derivative

\frac{\partial L}{\partial \mathbf{w}} = (\hat{y} - y)\mathbf{x}

(remarkably clean!)

◆ Why It Works — First-Principles Derivation (Logistic Regression & Classification Boundaries)

Why Integration by Parts & Integrating Factors Work: Differentiating a product $u(x)v(x)$ gives $(uv)' = u'v + uv'$ . Integrating both sides and rearranging yields the Integration by Parts formula:

\int u\,\frac{dv}{dx}\,dx = u v - \int v\,\frac{du}{dx}\,dx

Similarly, for a linear first-order ODE $y' + P(x)y = Q(x)$ , multiplying by the integrating factor $\mu(x) = e^{\int P(x)\,dx}$ (which satisfies $\mu'(x) = P(x)\mu(x)$ ) collapses the left-hand side into an exact product derivative: $\dfrac{d}{dx}\bigl[\mu(x)y\bigr] = \mu(x)Q(x)$ .

Worked Example 7.A — Probability & Cross-Entropy Calculation

A medical AI model has learned weights $w = 0.8$ and bias $b = -2.4$ for predicting whether a tumor is malignant ( $y=1$ ) based on diameter $x$ in cm:
$z = 0.8x - 2.4, \quad \hat{y} = \sigma(z)$
(a) Find the predicted malignancy probability for a tumor of diameter $x = 4.5 \text{ cm}$ .
(b) If the actual diagnosis was malignant ( $y = 1$ ), calculate the binary cross-entropy loss $L$ .

Solution:
(a) Compute logit $z$ :
$z = 0.8(4.5) - 2.4 = 3.6 - 2.4 = 1.2$ .
Compute probability via Sigmoid:
$\hat{y} = \sigma(1.2) = \frac{1}{1 + e^{-1.2}} = \frac{1}{1 + 0.3012} = \frac{1}{1.3012} \approx 0.7685$ (76.85% probability).

(b) Binary cross-entropy with $y = 1$ :
$L = -\Big[ 1 \cdot \ln(0.7685) + 0 \cdot \ln(1 - 0.7685) \Big] = -\ln(0.7685) \approx -(-0.2633) = 0.2633$ .
Answer: (a) $\hat{y} \approx 0.769$ (76.9%), (b) Loss $L \approx 0.263$ .

Worked Example

B

— Logistic Regression & Classification Boundaries (Hard Application)

Problem (Hard): Consider a binary classification problem where the decision boundary is defined by the logistic function $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ , which represents the probability of a data point $\mathbf{x} = (x_1, x_2)$ belonging to class 1. The estimated model parameters are $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ . We are interested in the geometric shape of the classification boundary. The classification boundary occurs where $p(x) = 0.5$ . Find the equation of this boundary in the $x_1x_2$ -plane.

Step-by-Step Solution: The logistic regression model gives the probability of belonging to class 1 as $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ . The classification boundary is the set of points where $p(x) = 0.5$ . Setting $p(x) = 0.5$ , we have: $0.5 = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ This implies: $1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 2$ Therefore: $e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 1$ Taking the natural logarithm of both sides: $-(\beta_0 + \beta_1 x_1 + \beta_2 x_2) = \ln(1)$ Since $\ln(1) = 0$ , we get: $\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0$ This is the equation of the linear decision boundary. Substitute the given parameter values: $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ . $-3 + x_1 + (-2) \cdot x_2 = 0$ $-3 + x_1 - 2x_2 = 0$ We need to express this in the form $x_2 = f(x_1)$ . $2x_2 = x_1 - 3$ $x_2 = \frac{1}{2} x_1 - \frac{3}{2}$ This matches choice A.

مسألة (Hard): Consider a binary classification problem where the decision boundary is defined by the logistic function $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ , which represents the probability of a data point $\mathbf{x} = (x_1, x_2)$ belonging to class 1. The estimated model parameters are $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ . We are interested in the geometric shape of the classification boundary. The classification boundary occurs where $p(x) = 0.5$ . Find the equation of this boundary in the $x_1x_2$ -plane.

الحل خطوة بخطوة: The logistic regression model gives the probability of belonging to class 1 as $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ . The classification boundary is the set of points where $p(x) = 0.5$ . Setting $p(x) = 0.5$ , we have: $0.5 = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ This implies: $1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 2$ Therefore: $e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 1$ Taking the natural logarithm of both sides: $-(\beta_0 + \beta_1 x_1 + \beta_2 x_2) = \ln(1)$ Since $\ln(1) = 0$ , we get: $\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0$ This is the equation of the linear decision boundary. Substitute the given parameter values: $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ . $-3 + x_1 + (-2) \cdot x_2 = 0$ $-3 + x_1 - 2x_2 = 0$ We need to express this in the form $x_2 = f(x_1)$ . $2x_2 = x_1 - 3$ $x_2 = \frac{1}{2} x_1 - \frac{3}{2}$ This matches choice A.

Worked Example

C

— Logistic Regression & Classification Boundaries (Easy Application)

Problem (Easy): Consider a simple logistic regression model where the decision boundary is defined by the linear equation $w_1x_1 + w_2x_2 + b = 0$ . If the weights are $w_1 = 2$ and $w_2 = 3$ , and the bias is $b = -7$ , what is the equation of the decision boundary?

Step-by-Step Solution: The decision boundary in a logistic regression model is defined by the equation where the log-odds are zero. This corresponds to a linear combination of the input features and weights, plus the bias, being equal to zero. Given the weights $w_1 = 2$ and $w_2 = 3$ , and the bias $b = -7$ , the equation of the decision boundary is $w_1x_1 + w_2x_2 + b = 0$ . Substituting the given values, we get $2x_1 + 3x_2 + (-7) = 0$ , which simplifies to $2x_1 + 3x_2 - 7 = 0$ .

Worked Example

D

— Logistic Regression & Classification Boundaries (Medium Application)

Problem (Medium): Consider a simple logistic regression model where the decision boundary is defined by the equation $-1.5x_1 + 2x_2 + 5 = 0$ . For a data point $(x_1, x_2)$ , if $ -1.5x_1 + 2x_2 + 5 < 0 $, it is classified into Class 0. Otherwise, it is classified into Class 1. Which of the following points lies on the decision boundary?

Step-by-Step Solution: The decision boundary is defined by the equation $-1.5x_1 + 2x_2 + 5 = 0$ . To find which point lies on the boundary, we substitute the coordinates of each point into the equation. For point $(-1, 1)$ : $-1.5(-1) + 2(1) + 5 = 1.5 + 2 + 5 = 8.5$ , which is not 0. For point $(2, -2)$ : $-1.5(2) + 2(-2) + 5 = -3 - 4 + 5 = -2$ , which is not 0. For point $(0, -2.5)$ : $-1.5(0) + 2(-2.5) + 5 = 0 - 5 + 5 = 0$ . This point lies on the boundary. For point $(3, 0.5)$ : $-1.5(3) + 2(0.5) + 5 = -4.5 + 1 + 5 = 1.5$ , which is not 0. Therefore, the point $(0, -2.5)$ lies on the decision boundary.

⚠️ Common Exam Traps — Logistic Regression & Classification Boundaries

1. Adding $+C$ Only After Exponentiating in Separable ODEs: In $\dfrac{dy}{dx} = 3xy$ , integrating gives $\ln|y| = \tfrac{3}{2}x^2 + C \implies y = Ae^{3x^2/2}$ where $A = \pm e^C$ . Writing $y = e^{3x^2/2} + C$ is a fatal algebraic error that forfeits all follow-up marks on Mathematics for Artificial Intelligence & Machine Learning.

2. Forgetting to Transform Limits in $u$ -Substitution: In definite integrals $\int_a^b f(g(x))g'(x)\,dx$ , you must replace the limits $x = a, b$ with $u = g(a), g(b)$ immediately.

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Logistic Regression & Classification Boundaries)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Writing $\ln\|y\| = x^2 + C \implies y = e^{x^2} + C$	Exponentiation distributes over addition as multiplication ( $e^{a+b} = e^a e^b$ ), not addition.	Write $y = e^{x^2 + C} = Ae^{x^2}$ and solve for $A$ using the initial condition.
Leaving original $x$ -limits on a $u$ -substituted definite integral	Evaluates the antiderivative $F(u)$ at $x$ -values instead of $u(x)$ -values.	Convert bounds immediately: $u_{\text{low}} = u(a)$ and $u_{\text{high}} = u(b)$ .
Computing total distance as $\left\|\int_{t_1}^{t_2} v(t)\,dt\right\|$	Cancels forward and backward motion (net displacement) instead of accumulating total path length.	Split the integral at every root where $v(t) = 0$ and integrate $\|v(t)\|$ .

▲ Exam Pacing & Verification Strategy — Logistic Regression & Classification Boundaries

Quick Verification Trick (Mathematics for Artificial Intelligence & Machine Learning): For any indefinite integral $\int f(x)\,dx = F(x) + C$ , differentiate your answer $F'(x)$ mentally—especially the coefficient $1/k$ from chain-rule reversal $\int e^{kx}\,dx = \tfrac{1}{k}e^{kx} + C$ —before moving on.

[Practice 1 · Hard] The sigmoid activation function is

\sigma(z) = \frac{1}{1 + e^{-z}}

. If

\sigma(z_0) = 0.40

, what is the value of the derivative

\sigma'(z_0)

💡 Hint 1 of 3

The derivative of sigmoid satisfies

\sigma'(z) = \sigma(z)(1 - \sigma(z))

💡 Hint 2 of 3

Substituting

\sigma(z_0) = 0.40

\sigma'(z_0) = (0.40)(1 - 0.40)

. Substitute and simplify:

\sigma'(z_0) = (0.40)(0.60)

💡 Hint 3 of 3

Compute the final value from:

(0.40)(0.60)

The derivative of sigmoid satisfies

\sigma'(z) = \sigma(z)(1 - \sigma(z))

. Substituting

\sigma(z_0) = 0.40

\sigma'(z_0) = (0.40)(1 - 0.40) = (0.40)(0.60) = 0.2400

[Practice 2 · Easy] A fitted simple linear regression model is

\hat{y} = 7x + 18

. What is the predicted target value

\hat{y}

when the input feature is

x = 4

💡 Hint 1 of 3

Substitute

x = 4

into the regression equation:

\hat{y} = 7(4) + 18

💡 Hint 2 of 3

Substitute and simplify:

\hat{y} = 28 + 18

💡 Hint 3 of 3

Compute the final value from:

28 + 18

Substitute

x = 4

into the regression equation:

\hat{y} = 7(4) + 18 = 28 + 18 = 46

[Practice 3 · Medium] A binary classification model uses a logistic sigmoid function

\sigma(z) = \frac{1}{1 + e^{-z}}

. The decision boundary is defined by

\sigma(z) = 0.5

, where

z = w_1 x_1 + w_0

. For a dataset with two features

x_1

and

x_2

, the model's linear component is

z = 2x_1 - x_2 + b

. If the classification boundary passes through the point

(x_1, x_2) = (3, 1)

, what is the value of the bias

b

💡 Hint 1 of 3

The logistic sigmoid function is

\sigma(z) = \frac{1}{1 + e^{-z}}

. The decision boundary is where

\sigma(z) = 0.5

. Setting

\sigma(z) = 0.5

gives

\frac{1}{1 + e^{-z}} = 0.5

, which implies

1 = 0.5(1 + e^{-z})

, so

2 = 1 + e^{-z}

, and

e^{-z} = 1

. This occurs when

-z = 0

, or

z = 0

. Therefore, the decision boundary is defined by

z = 0

💡 Hint 2 of 3

The linear component of the model is given by

z = w_1 x_1 + w_2 x_2 + b

. In this problem,

z = 2x_1 - x_2 + b

. Since the decision boundary is

z = 0

, we have

2x_1 - x_2 + b = 0

💡 Hint 3 of 3

Use the resulting form to finish:

5 + b

The logistic sigmoid function is

\sigma(z) = \frac{1}{1 + e^{-z}}

. The decision boundary is where

\sigma(z) = 0.5

. Setting

\sigma(z) = 0.5

gives

\frac{1}{1 + e^{-z}} = 0.5

, which implies

1 = 0.5(1 + e^{-z})

, so

2 = 1 + e^{-z}

, and

e^{-z} = 1

. This occurs when

-z = 0

, or

z = 0

. Therefore, the decision boundary is defined by

z = 0

. The linear component of the model is given by

z = w_1 x_1 + w_2 x_2 + b

. In this problem,

z = 2x_1 - x_2 + b

. Since the decision boundary is

z = 0

, we have

2x_1 - x_2 + b = 0

. The classification boundary passes through the point

(x_1, x_2) = (3, 1)

. Substituting these values into the equation for the decision boundary:

2(3) - 1 + b = 0

6 - 1 + b = 0

5 + b = 0

b = -5

[Practice 4 · Hard] For a bivariate feature dataset,

\text{Cov}(x, y) = 24

\text{Var}(x) = 8

\bar{x} = 5

, and

\bar{y} = 30

. Compute the OLS regression slope

\hat{\beta}_1

and intercept

\hat{\beta}_0

💡 Hint 1 of 3

The OLS slope is

\hat{\beta}_1 = \frac{\text{Cov}(x, y)}{\text{Var}(x)} = \frac{24}{8}

💡 Hint 2 of 3

The intercept is:

\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x}

. Substitute and simplify:

\hat{\beta}_0 = 30 - (3)(5)

💡 Hint 3 of 3

Compute the final value from:

30 - 15

The OLS slope is

\hat{\beta}_1 = \frac{\text{Cov}(x, y)}{\text{Var}(x)} = \frac{24}{8} = 3

. The intercept is

\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x} = 30 - (3)(5) = 30 - 15 = 15

[Practice 5 · Easy] In evaluating a binary classifier, the confusion matrix shows

TP = 45

TN = 40

FP = 10

, and

FN = 10

. Compute the overall classification accuracy.

💡 Hint 1 of 3

Accuracy is the ratio of correct predictions to total samples:

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}

💡 Hint 2 of 3

Substitute and simplify:

\text{Accuracy} = \frac{45 + 40}{105}

💡 Hint 3 of 3

Compute the final value from:

\frac{45 + 40}{105}

Accuracy is the ratio of correct predictions to total samples:

\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{45 + 40}{105} = \frac{85}{105} = 81.0\%

📝 Chapter Quiz

Questions Level

Dimensionality Reduction, Covariance & PCA

🤖 AI Math Unit 8 — Dimensionality Reduction & PCA Applied STEM · Curse of dimensionality in high-dimensional data · Sample covariance matrix Σ · Eigenvalues λ and Eigenvectors v · Principal Component Analysis (PCA) projection · Explained variance ratio and data compression

Real datasets frequently have hundreds or thousands of correlated features. Principal Component Analysis (PCA) identifies the orthogonal axes of maximum variance, compressing high-dimensional data while preserving essential information.

الشرح الرياضي (تقليل الأبعاد ومصفوفة التباين المشترك وتحليل المكونات الرئيسية (PCA)):

▲ Theorem 8.1 — PCA via Spectral Decomposition

Let

\mathbf{X} \in \mathbb{R}^{N \times d}

be a zero-centered data matrix (mean

\mathbf{\bar{x}} = \mathbf{0}

).
1. The empirical covariance matrix is

\mathbf{\Sigma} = \frac{1}{N-1} \mathbf{X}^T \mathbf{X} \in \mathbb{R}^{d \times d}

.
2. Because

\mathbf{\Sigma}

is symmetric positive semi-definite, it admits an orthonormal eigen-decomposition:

\mathbf{\Sigma} \mathbf{v}_i = \lambda_i \mathbf{v}_i, \quad \lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_d \ge 0

3. The first principal component

\mathbf{v}_1

is the unit eigenvector associated with the largest eigenvalue

\lambda_1

, maximizing the variance of the projected data.

● Deep Conceptual Intuition — Dimensionality Reduction, Covariance & PCA

Coordinate grid showing geometric action of a linear transformation mapping △ABC to its image △A'B'C'.

⊙ Explorer · PCA: Covariance Point Cloud, Principal Axis

v_{1}

& Projections

interactive

Live numerical table for PCA: Covariance Point Cloud, Principal Axis $v_{1}$ & Projections
Parameter / Metric	Value

Candidate Axis Angle $\theta$ (°) = 35

Cloud Correlation $\rho =$ 0.8

Projected Variance along

u(\theta)

Var_

\theta = 1.752

Optimal First Eigenvector

\theta

\theta * = 45.0^\circ

Explained Variance Ratio

EVR_{1} = 90.0\%

Rotate unit direction vector $u(\theta)$ through the centered 2D data cloud. PCA chooses the angle $\theta$ * that maximizes the variance of the orthogonal projections (and minimizes perpendicular reconstruction error).

Dimensionality Reduction, Covariance & PCA — Mathematical Diagram (2) — illustrating key geometric and analytical relationships in Dimensionality Reduction, Covariance & PCA.

Covariance Entry

\text{Cov}(X, Y) = \frac{1}{N-1}\sum (x_i - \bar{x})(y_i - \bar{y})

Characteristic Equation

\det(\mathbf{\Sigma} - \lambda \mathbf{I}) = 0

Explained Variance

\text{Ratio}_k = \frac{\lambda_k}{\sum_{j=1}^d \lambda_j}

Projection

\mathbf{z} = \mathbf{V}_k^T \mathbf{x} \in \mathbb{R}^k

(reduced representation)

◆ Why It Works — First-Principles Derivation (Dimensionality Reduction, Covariance & PCA)

\mathbf{u}\cdot\mathbf{v} = c(\mathbf{v}\cdot\mathbf{v}) + 0 \implies c = \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{v}\|^2}

Worked Example 8.A — Finding Principal Components of 2D Data

A 2D centered dataset has sample covariance matrix:
$\mathbf{\Sigma} = \begin{pmatrix} 4 & 2 \\ 2 & 1 \end{pmatrix}$
(a) Find the eigenvalues $\lambda_1$ and $\lambda_2$ .
(b) Find the unit eigenvector $\mathbf{v}_1$ corresponding to the principal component.
(c) What percentage of total variance is explained by the first principal component?

Solution:
(a) Characteristic equation $\det(\mathbf{\Sigma} - \lambda\mathbf{I}) = 0$ :
$\det \begin{pmatrix} 4 - \lambda & 2 \\ 2 & 1 - \lambda \end{pmatrix} = (4 - \lambda)(1 - \lambda) - (2)(2) = 0$
$\lambda^2 - 5\lambda + 4 - 4 = 0 \implies \lambda^2 - 5\lambda = 0 \implies \lambda(\lambda - 5) = 0$ .
Eigenvalues are $\lambda_1 = 5$ and $\lambda_2 = 0$ .

(b) For $\lambda_1 = 5$ :
$(\mathbf{\Sigma} - 5\mathbf{I})\mathbf{v} = \begin{pmatrix} -1 & 2 \\ 2 & -4 \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} \implies -v_1 + 2v_2 = 0 \implies v_1 = 2v_2$ .
A directional vector is $\begin{pmatrix} 2 \\ 1 \end{pmatrix}$ . Normalizing to unit length:
$\|\mathbf{v}\| = \sqrt{2^2 + 1^2} = \sqrt{5} \implies \mathbf{v}_1 = \frac{1}{\sqrt{5}}\begin{pmatrix} 2 \\ 1 \end{pmatrix} = \begin{pmatrix} 2/\sqrt{5} \\ 1/\sqrt{5} \end{pmatrix} \approx \begin{pmatrix} 0.894 \\ 0.447 \end{pmatrix}$ .

(c) Total variance = $\lambda_1 + \lambda_2 = 5 + 0 = 5$ .
Percentage explained = $\frac{5}{5} = 100\%$ . (The data lies perfectly along a 1D line in 2D space!).
Answer: $\lambda_1 = 5, \, \lambda_2 = 0$ ; $\mathbf{v}_1 = [0.894, 0.447]^T$ ; 100% variance explained.

A 2D centered dataset has sample covariance المصفوفة:
$\mathbf{\Sigma} = \begin{pmatrix} 4 & 2 \\ 2 & 1 \end{pmatrix}$
(a) أوجد القيم الذاتية $\lambda_1$ and $\lambda_2$ .
(b) أوجد unit eigenvector $\mathbf{v}_1$ corresponding to the principal component.
(c) What percentage of total variance is explained by the first principal component?

Solution:
(a) Characteristic المعادلة $\det(\mathbf{\Sigma} - \lambda\mathbf{I}) = 0$ :
$\det \begin{pmatrix} 4 - \lambda & 2 \\ 2 & 1 - \lambda \end{pmatrix} = (4 - \lambda)(1 - \lambda) - (2)(2) = 0$
$\lambda^2 - 5\lambda + 4 - 4 = 0 \implies \lambda^2 - 5\lambda = 0 \implies \lambda(\lambda - 5) = 0$ .
Eigenvalues are $\lambda_1 = 5$ and $\lambda_2 = 0$ .

(b) For $\lambda_1 = 5$ :
$(\mathbf{\Sigma} - 5\mathbf{I})\mathbf{v} = \begin{pmatrix} -1 & 2 \\ 2 & -4 \end{pmatrix} \begin{pmatrix} v_1 \\ v_2 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \end{pmatrix} \implies -v_1 + 2v_2 = 0 \implies v_1 = 2v_2$ .
A directional المتجه is $\begin{pmatrix} 2 \\ 1 \end{pmatrix}$ . Normalizing to unit length:
$\|\mathbf{v}\| = \sqrt{2^2 + 1^2} = \sqrt{5} \implies \mathbf{v}_1 = \frac{1}{\sqrt{5}}\begin{pmatrix} 2 \\ 1 \end{pmatrix} = \begin{pmatrix} 2/\sqrt{5} \\ 1/\sqrt{5} \end{pmatrix} \approx \begin{pmatrix} 0.894 \\ 0.447 \end{pmatrix}$ .

(c) Total variance = $\lambda_1 + \lambda_2 = 5 + 0 = 5$ .
Percentage explained = $\frac{5}{5} = 100\%$ . (The data lies perfectly along a 1D line in 2D space!).
Answer: $\lambda_1 = 5, \, \lambda_2 = 0$ ; $\mathbf{v}_1 = [0.894, 0.447]^T$ ; 100% variance explained.

Worked Example

B

— Dimensionality Reduction, Covariance & PCA (Easy Application)

Problem (Easy): Consider a 2D data point $\mathbf{v} = \begin{bmatrix} 3 \\ -4 \end{bmatrix}$ . What is the squared Euclidean norm (or squared magnitude) of this vector?

Step-by-Step Solution: The squared Euclidean norm of a vector $\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix}$ is given by $\|\mathbf{v}\|^2 = v_1^2 + v_2^2$ . For the given vector $\mathbf{v} = \begin{bmatrix} 3 \\ -4 \end{bmatrix}$ , the squared Euclidean norm is $(3)^2 + (-4)^2 = 9 + 16 = 25$ .

Worked Example

C

— Dimensionality Reduction, Covariance & PCA (Medium Application)

Problem (Medium): A binary classifier $p(t)$ produces the confusion matrix below with $\text{TP} = 68$ , $\text{FP} = 10$ , $\text{FN} = 48$ , and $\text{TN} = 80$ . Determine the total number of correct classifications $q = \text{TP} + \text{TN}$ .

Step-by-Step Solution: Summing $\text{TP} + \text{TN} = 68 + 80 = 148$ .

Worked Example

D

— Dimensionality Reduction, Covariance & PCA (Hard Application)

Problem (Hard): Consider a 2D dataset represented by the points $A = (1, 2)$ , $B = (4, 2)$ , and $C = (2, 5)$ . If this dataset undergoes a linear transformation represented by the matrix $M = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}$ , what are the coordinates of the transformed points $A'$ , $B'$ , and $C'$ ?

Step-by-Step Solution: The transformation of a point $\mathbf{x}$ by a matrix $M$ is given by $M \mathbf{x}$ . We apply the matrix $M = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}$ to each point: For point $A = (1, 2)$ , the transformed point $A'$ is: $\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \end{pmatrix} = \begin{pmatrix} 2 \times 1 + 0 \times 2 \\ 0 \times 1 + 1 \times 2 \end{pmatrix} = \begin{pmatrix} 2 \\ 2 \end{pmatrix}$ So, $A' = (2, 2)$ . For point $B = (4, 2)$ , the transformed point $B'$ is: $\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 4 \\ 2 \end{pmatrix} = \begin{pmatrix} 2 \\ 2 \end{pmatrix}$ So, $B' = (8, 2)$ . For point $C = (2, 5)$ , the transformed point $C'$ is: $\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 5 \end{pmatrix} = \begin{pmatrix} 2 \times 2 + 0 \times 5 \\ 0 \times 2 + 1 \times 5 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}$ So, $C' = (4, 5)$ . Therefore, the transformed points are $A' = (2, 2)$ , $B' = (8, 2)$ , and $C' = (4, 5)$ .

⚠️ Common Exam Traps — Dimensionality Reduction, Covariance & PCA

Error Diagnosis — Common Student Traps vs. Exam-Board Method (Dimensionality Reduction, Covariance & PCA)

Common Student Trap	Why It Fails Mathematically	Correct Exam-Board Method
Computing angle at $B$ using $\overrightarrow{AB}\cdot\overrightarrow{BC}$	Vectors are head-to-tail instead of tail-to-tail, giving $180^\circ - \theta$ .	Reverse $\overrightarrow{AB}$ so both emanate from $B$ : use $\overrightarrow{BA}\cdot\overrightarrow{BC}$ .
Using the same parameter $t$ for two intersecting 3D lines $\mathbf{r}_1(t) = \mathbf{r}_2(t)$	Tests for a simultaneous collision at time $t$ rather than geometric intersection of paths.	Assign distinct parameters $\lambda$ and $\mu$ to the two lines before equating components.
Assuming $(AB)^{-1} = A^{-1}B^{-1}$	Multiplying $(AB)(A^{-1}B^{-1})$ does not cancel inner factors when $AB \ne BA$ .	Use the shoe-sock reversal law: $(AB)^{-1} = B^{-1}A^{-1}$ .

▲ Exam Pacing & Verification Strategy — Dimensionality Reduction, Covariance & PCA

[Practice 1 · Easy] Consider a dataset of observations

\{x_i\}_{i=1}^n

where each

x_i

is a 2-dimensional vector. The sample covariance matrix for this dataset is given by:

S = \begin{pmatrix} 4 & -2 \\ -2 & 1 \end{pmatrix}

Principal Component Analysis (PCA) is applied to this dataset. The first principal component corresponds to the direction with the largest variance. Let the direction vector of the first principal component be

\mathbf{v}_1 = \begin{pmatrix} a \\ b \end{pmatrix}

. The variance explained by this principal component is given by the corresponding eigenvalue

\lambda_1

. If

\mathbf{v}_1

is a unit vector, what is the value of

a^2+b^2

💡 Hint 1 of 3

The first principal component direction vector

\mathbf{v}_1

is an eigenvector of the covariance matrix

S

💡 Hint 2 of 3

The variance explained by this component is the corresponding eigenvalue

\lambda_1

. The problem states that

\mathbf{v}_1 = \begin{pmatrix} a \\ b \end{pmatrix}

is a unit vector.

💡 Hint 3 of 3

The specific values of

a

and

b

and the eigenvalues of

S

are not needed to answer this question, as it directly asks for the property of a unit vector.

The first principal component direction vector

\mathbf{v}_1

is an eigenvector of the covariance matrix

S

. The variance explained by this component is the corresponding eigenvalue

\lambda_1

. The problem states that

\mathbf{v}_1 = \begin{pmatrix} a \\ b \end{pmatrix}

is a unit vector. By definition, a unit vector has a magnitude of 1. The squared magnitude of a vector

\begin{pmatrix} a \\ b \end{pmatrix}

a^2 + b^2

. Therefore, if

\mathbf{v}_1

is a unit vector, its squared magnitude must be 1, which means

a^2 + b^2 = 1

. The specific values of

a

and

b

and the eigenvalues of

S

are not needed to answer this question, as it directly asks for the property of a unit vector.

[Practice 2 · Medium] Consider a dataset of

n

data points, each residing in

\mathbb{R}^D

. We are performing Principal Component Analysis (PCA) to reduce the dimensionality. The first three principal components capture

70\%

of the total variance, the first six capture

85\%

, and the first ten capture

92\%

. You are tasked with selecting a subset of principal components for a downstream machine learning model. If the model requires that at least

90\%

of the original variance be retained, and you want to use the minimum possible number of principal components, what is the dimensionality of the transformed data?

💡 Hint 1 of 3

The problem requires us to find the minimum number of principal components such that the cumulative explained variance is at least

90\%

. The provided information states: - The first 3 principal components capture

70\%

of the variance.

💡 Hint 2 of 3

- The first 6 principal components capture

85\%

of the variance. - The first 10 principal components capture

92\%

of the variance.

💡 Hint 3 of 3

Therefore, the minimum number of principal components required to retain at least

90\%

of the variance is 10. When PCA reduces the dimensionality of data from

\mathbb{R}^D

\mathbb{R}^k

using the top

k

principal components, the dimension of each transformed data point is

k

. In this case,

k

The problem requires us to find the minimum number of principal components such that the cumulative explained variance is at least

90\%

. The provided information states: - The first 3 principal components capture

70\%

of the variance. - The first 6 principal components capture

85\%

of the variance. - The first 10 principal components capture

92\%

of the variance. We need to find the smallest

k

such that the cumulative variance captured by the first

k

principal components is

\ge 90\%

. Checking the given values: - For

k = 3

, variance captured =

70\%

, which is less than

90\%

. - For

k = 6

, variance captured =

85\%

, which is less than

90\%

. - For

k = 10

, variance captured =

92\%

, which is greater than or equal to

90\%

. Therefore, the minimum number of principal components required to retain at least

90\%

of the variance is 10. When PCA reduces the dimensionality of data from

\mathbb{R}^D

\mathbb{R}^k

using the top

k

principal components, the dimension of each transformed data point is

k

. In this case,

k=10

[Practice 3 · Hard] In training a language model token predictor, a binary token occurs

k = 6

times across

n = 20

independent trials. What is the Maximum Likelihood Estimate

\hat{p}_{\text{MLE}}

of the token probability?

💡 Hint 1 of 3

For a Bernoulli likelihood

L(p) = p^k (1-p)^{n-k}

, the log-likelihood is

\ln L = k\ln p + (n-k)\ln(1-p)

💡 Hint 2 of 3

Setting derivative to zero:

\frac{k}{p} - \frac{n-k}{1-p} = 0

. Substitute and simplify:

\hat{p} = \frac{k}{n}

💡 Hint 3 of 3

Evaluate the simplified expression:

\hat{p} = \frac{6}{20}

For a Bernoulli likelihood

L(p) = p^k (1-p)^{n-k}

, the log-likelihood is

\ln L = k\ln p + (n-k)\ln(1-p)

. Setting derivative to zero:

\frac{k}{p} - \frac{n-k}{1-p} = 0 \implies \hat{p} = \frac{k}{n} = \frac{6}{20} = 0.300

[Practice 4 · Easy] An Ordinary Least Squares linear model

\hat{p}(t) = 2t + 11

is evaluated at

t = 3

. Determine the predicted response

\hat{p}(3)

💡 Hint 1 of 3

Start from the given mathematical expression in the prompt:

\hat{p}(t) = 2t + 11

💡 Hint 2 of 3

Write out the unsimplified expression:

2(3) + 11

💡 Hint 3 of 3

Perform the arithmetic in

2(3) + 11

to find the final value.

Evaluating

2(3) + 11 = 17

[Practice 5 · Medium] A machine learning diagnostic test has sensitivity

P(\text{Positive} | \text{Defect}) = 0.90

. If the defect rate in production is

P(\text{Defect}) = 0.10

, find the joint probability of a defect and a positive test result

P(\text{Defect} \cap \text{Positive})

💡 Hint 1 of 3

By the multiplication rule of conditional probability:

P(\text{Defect} \cap \text{Positive}) = P(\text{Defect}) \times P(\text{Positive} | \text{Defect})

💡 Hint 2 of 3

Substitute and simplify:

P(\text{Defect} \cap \text{Positive}) = 0.10 \times 0.90

💡 Hint 3 of 3

Compute the final value from:

0.10 \times 0.90

By the multiplication rule of conditional probability:

P(\text{Defect} \cap \text{Positive}) = P(\text{Defect}) \times P(\text{Positive} | \text{Defect}) = 0.10 \times 0.90 = 0.0900

📝 Chapter Quiz

Questions Level

Practice set

Fifty applied mathematics problems for AI & Data Science spanning high-dimensional vectors and cosine similarity, neural layer matrix transformations, multivariable calculus and gradient descent optimization, conditional probability and Bayes' theorem, probability distributions and maximum likelihood estimation (MLE), ordinary least squares linear regression, logistic regression and cross-entropy loss, and covariance matrices and Principal Component Analysis (PCA).

In anomaly detection, anomaly prior $P(A) = 0.02$ , detection sensitivity $P(+|A) = 0.95$ , and false alarm rate $P(+|\neg A) = 0.05$ . By Bayes' theorem, what is the posterior probability $P(A|+)$ that an alarm is a true anomaly?

Advanced

By Bayes' Rule: $P(A|+) = \frac{P(+|A)P(A)}{P(+|A)P(A) + P(+|\neg A)P(\neg A)} = \frac{0.95 \times 0.02}{0.95(0.02) + 0.05(0.98)} = \frac{0.0190}{0.0680} = 0.279$ .

Final Answer: $0.279$

Consider a dataset of $n$ data points, each residing in $\mathbb{R}^D$ . We are performing Principal Component Analysis (PCA) to reduce the dimensionality. The first three principal components capture $70\%$ of the total variance, the first six capture $85\%$ , and the first ten capture $92\%$ . You are tasked with selecting a subset of principal components for a downstream machine learning model. If the model requires that at least $90\%$ of the original variance be retained, and you want to use the minimum possible number of principal components, what is the dimensionality of the transformed data?

Advanced

The problem requires us to find the minimum number of principal components such that the cumulative explained variance is at least $90\%$ . The provided information states:

- The first 3 principal components capture $70\%$ of the variance.
- The first 6 principal components capture $85\%$ of the variance.
- The first 10 principal components capture $92\%$ of the variance.

We need to find the smallest $k$ such that the cumulative variance captured by the first $k$ principal components is $\ge 90\%$ .

Checking the given values:
- For $k = 3$ , variance captured = $70\%$ , which is less than $90\%$ .
- For $k = 6$ , variance captured = $85\%$ , which is less than $90\%$ .
- For $k = 10$ , variance captured = $92\%$ , which is greater than or equal to $90\%$ .

Therefore, the minimum number of principal components required to retain at least $90\%$ of the variance is 10. When PCA reduces the dimensionality of data from $\mathbb{R}^D$ to $\mathbb{R}^k$ using the top $k$ principal components, the dimension of each transformed data point is $k$ . In this case, $k=10$ .

Given $n = 12$ independent observations from a normal distribution with sum $\sum_{i=1}^{12} x_i = 36$ , what is the Maximum Likelihood Estimate (MLE) $\hat{\mu}_{\text{MLE}}$ ?

Intermediate

The log-likelihood $\ln L(\mu) = -\frac{n}{2}\ln(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum (x_i - \mu)^2$ . Setting $\frac{\partial \ln L}{\partial \mu} = 0$ yields the sample mean $\hat{\mu} = \frac{1}{n}\sum x_i = \frac{36}{12} = 3$ .

Final Answer: $\hat{\mu}_{\text{MLE}} = 3$

Consider the scalar loss function $L(w) = 6w^2 - 8w + 5$ . Find the analytical gradient $\nabla L(w)$ evaluated at $w = 10$ .

Intermediate

Differentiate with respect to $w$ : $\frac{dL}{dw} = 2(6)w - 8 = 12w - 8$ . At $w = 10$ : $\nabla L = 12(10) - 8 = 120 - 8 = 112$ .

Final Answer: $\nabla L(10) = 112$

Key Mathematical Principle & Verification: In exam revision, always verify that the final result satisfies all domain restrictions, boundary conditions, and the original governing equation by direct substitution into the prompt.

For a bivariate feature dataset, $\text{Cov}(x, y) = 36$ , $\text{Var}(x) = 12$ , $\bar{x} = 5$ , and $\bar{y} = 35$ . Compute the OLS regression slope $\hat{\beta}_1$ and intercept $\hat{\beta}_0$ .

Advanced

The OLS slope is $\hat{\beta}_1 = \frac{\text{Cov}(x, y)}{\text{Var}(x)} = \frac{36}{12} = 3$ . The intercept is $\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x} = 35 - (3)(5) = 35 - 15 = 20$ .

Final Answer: $\hat{\beta}_1 = 3, \; \hat{\beta}_0 = 20$

A spam classification model identifies $TP = 120$ genuine spam emails and $FP = 40$ legitimate emails falsely flagged as spam. What is the precision of this model?

Intermediate

Precision is the fraction of predicted positive instances that are truly positive: $\text{Precision} = \frac{TP}{TP + FP} = \frac{120}{120 + 40} = \frac{120}{160} = 75.0\%$ .

Final Answer: $75.0\%$

Consider a binary classification problem where the decision boundary is defined by the logistic function $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ , which represents the probability of a data point $\mathbf{x} = (x_1, x_2)$ belonging to class 1. The estimated model parameters are $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ . We are interested in the geometric shape of the classification boundary. The classification boundary occurs where $p(x) = 0.5$ . Find the equation of this boundary in the $x_1x_2$ -plane.

Advanced

The logistic regression model gives the probability of belonging to class 1 as $p(x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$ . The classification boundary is the set of points where $p(x) = 0.5$ .

Setting $p(x) = 0.5$ , we have:
$0.5 = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}$

This implies:
$1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 2$

Therefore:
$e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 1$

Taking the natural logarithm of both sides:
$-(\beta_0 + \beta_1 x_1 + \beta_2 x_2) = \ln(1)$

Since $\ln(1) = 0$ , we get:
$\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0$

This is the equation of the linear decision boundary.

Substitute the given parameter values: $\beta_0 = -3$ , $\beta_1 = 1$ , and $\beta_2 = -2$ .

$-3 + x_1 + (-2) \cdot x_2 = 0$

$-3 + x_1 - 2x_2 = 0$

We need to express this in the form $x_2 = f(x_1)$ .

$2x_2 = x_1 - 3$

$x_2 = \frac{1}{2} x_1 - \frac{3}{2}$

A 2D linear transformation $T$ is defined by the matrix $M = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}$ . Consider a triangle with vertices $A = (1, 0)$ , $B = (0, 1)$ , and $C = (1, 1)$ . This triangle is transformed by $T$ to a new triangle with vertices $A'$ , $B'$ , and $C'$ . The area of the transformed triangle $A'B'C'$ is then passed through a ReLU activation function, $f(x) = \max(0, x)$ . What is the final output value?

Advanced

The area of the original triangle with vertices $A = (1, 0)$ , $B = (0, 1)$ , and $C = (1, 1)$ can be calculated. We can use the determinant formula for the area of a triangle given its vertices $(x_1, y_1), (x_2, y_2), (x_3, y_3)$ : Area = $\frac{1}{2} |x_1(y_2 - y_3) + x_2(y_3 - y_1) + x_3(y_1 - y_2)|$ .
Plugging in the coordinates: Area = $\frac{1}{2} |1(1 - 1) + 0(1 - 0) + 1(0 - 1)| = \frac{1}{2} |0 + 0 - 1| = \frac{1}{2}$ .

A linear transformation represented by a matrix $M$ scales the area of a 2D shape by the absolute value of the determinant of the matrix $M$ . The determinant of $M = \begin{pmatrix} 3 & 1 \\ 1 & 3 \end{pmatrix}$ is $\det(M) = (3 \times 3) - (1 \times 1) = 9 - 1 = 8$ . The absolute value is $|8| = 8$ .
Therefore, the area of the transformed triangle $A'B'C'$ is the original area multiplied by the determinant: Area( $A'B'C'$ ) = $\frac{1}{2} \times 8 = 4$ .
This area is then passed through a ReLU activation function $f(x) = \max(0, x)$ . Since the area is 4, which is positive, the ReLU function will return the value itself: $f(4) = \max(0, 4) = 4$ .
Thus, the final output value is 4.

Final Answer: $4$

The sigmoid activation function is $\sigma(z) = \frac{1}{1 + e^{-z}}$ . If $\sigma(z_0) = 1.20$ , what is the value of the derivative $\sigma'(z_0)$ ?

Advanced

The derivative of sigmoid satisfies $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ . Substituting $\sigma(z_0) = 1.20$ : $\sigma'(z_0) = (1.20)(1 - 1.20) = (1.20)(-0.20) = -0.2400$ .

Final Answer: $-0.2400$

A 3D dataset has covariance matrix eigenvalues $\lambda_1 = 62$ , $\lambda_2 = 24$ , and $\lambda_3 = 14$ . What percentage of total variance is explained by the first principal component $PC_1$ ?

Intermediate

Total variance is $\sum \lambda_i = 62 + 24 + 14 = 100$ . The proportion explained by $PC_1$ is $\frac{\lambda_1}{\sum \lambda_i} = \frac{62}{100} = 62.0\%$ .

Final Answer: $62.0\%$

Two document embeddings $\mathbf{a} = [5, 10]^T$ and $\mathbf{b} = [10, 5]^T$ are compared. If $\|\mathbf{a}\| = \|\mathbf{b}\| = \sqrt{125}$ , calculate their exact cosine similarity.

Advanced

Dot product $\mathbf{a} \cdot \mathbf{b} = (5)(10) + (10)(5) = 100$ . Product of norms $\|\mathbf{a}\| \|\mathbf{b}\| = 125$ . Cosine similarity = $\frac{100}{125} = 0.800$ .

Final Answer: $0.800$

Consider a dataset of $n$ data points, where each data point is a vector in $\mathbb{R}^{D}$ . A non-linear dimensionality reduction technique maps these data points to a lower-dimensional space $\mathbb{R}^{d}$ . The mapping function for a single data point $\mathbf{x} \in \mathbb{R}^{D}$ is given by $f(\mathbf{x}) = \phi(\mathbf{x})$ , where $\phi: \mathbb{R}^{D} \to \mathbb{R}^{d}$ is a function designed to preserve local neighborhood structures. Suppose the mapping $\phi$ is defined by an explicit feature map $\phi(\mathbf{x}) = [\mathbf{x}^T \mathbf{x}, \mathbf{x}^T \mathbf{x}, \dots, \mathbf{x}^T \mathbf{x}]^T$ where the vector $\mathbf{x} \in \mathbb{R}^{10}$ and $d$ is the dimension of the embedded space. If the goal is to embed these points into a space such that pairwise Euclidean distances are preserved as closely as possible under a specific distortion, and it is found that $d=15$ yields optimal results for this mapping under a certain constraint, determine the effective rank of the feature map's image space.

Advanced

The feature map is defined as $\phi(\mathbf{x}) = [\mathbf{x}^T \mathbf{x}, \mathbf{x}^T \mathbf{x}, \dots, \mathbf{x}^T \mathbf{x}]^T$ , where $\mathbf{x} \in \mathbb{R}^{10}$ . The term $\mathbf{x}^T \mathbf{x}$ is the squared Euclidean norm of $\mathbf{x}$ , denoted as $\|\mathbf{x}\|^2$ . This is a scalar value. Thus, the mapping $\phi$ produces a vector where each component is the same scalar value, $\|\mathbf{x}\|^2$ . Let $k = \|\mathbf{x}\|^2$ . Then $\phi(\mathbf{x}) = [k, k, \dots, k]^T$ in $\mathbb{R}^{d}$ . This means that the image of the mapping $\phi$ is a subspace spanned by a single vector. For example, if $d=15$ , then $\phi(\mathbf{x}) = [k, k, \dots, k]^T$ which can be written as $k \cdot [1, 1, \dots, 1]^T$ . The vector $[1, 1, \dots, 1]^T$ forms a basis for the image space. Therefore, the dimension of the image space, which is equivalent to its rank, is 1, regardless of the value of $d$ (as long as $d \ge 1$ ). The problem states that $d=15$ , but the effective rank of the image of this specific feature map is always 1, as all output vectors lie on a single line passing through the origin.

A medical test for a rare disease has a 99\% accuracy rate in detecting the disease when it is present (True Positive Rate) and a 95\% accuracy rate in identifying individuals without the disease (True Negative Rate). If the prevalence of the disease in the population is 0.5\%, what is the probability that a person who tests positive actually has the disease?

Basic

Let D be the event that a person has the disease, and P be the event that the person tests positive.
We are given:
P(P|D) = 0.99 (True Positive Rate)
P(not P|not D) = 0.95 (True Negative Rate), which means P(P|not D) = 1 - 0.95 = 0.05 (False Positive Rate)
P(D) = 0.005 (Prevalence of the disease)
From P(D), we know P(not D) = 1 - P(D) = 1 - 0.005 = 0.995.
We want to find P(D|P), the probability that a person has the disease given a positive test result.
Using Bayes' Theorem:
P(D|P) = [P(P|D) * P(D)] / P(P)
To find P(P), we use the law of total probability:
P(P) = P(P|D) * P(D) + P(P|not D) * P(not D)
P(P) = (0.99 * 0.005) + (0.05 * 0.995)
P(P) = 0.00495 + 0.04975
P(P) = 0.0547
Now, substitute this back into Bayes' Theorem:
P(D|P) = (0.99 * 0.005) / 0.0547
P(D|P) = 0.00495 / 0.0547
P(D|P) \approx 0.09049
Converting to a percentage, this is approximately 9.05\%, which rounds to 9.1\%.

Final Answer: Approximately 9.1\%

Consider a dataset with $n$ data points $\{ \mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_n \}$ , where each $\mathbf{x}_i \in \mathbb{R}^p$ . The sample covariance matrix is given by $\mathbf{S} = \frac{1}{n-1} \sum_{i=1}^n (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^T$ , where $\bar{\mathbf{x}} = \frac{1}{n} \sum_{i=1}^n \mathbf{x}_i$ is the sample mean. Suppose PCA is performed on this dataset, resulting in principal components defined by the eigenvectors of $\mathbf{S}$ and corresponding variances given by the eigenvalues. Let $\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_p$ be the orthonormal eigenvectors of $\mathbf{S}$ with associated eigenvalues $\lambda_1 \ge \lambda_2 \ge \dots \ge \lambda_p$ . The first principal component captures the direction of maximum variance, $\lambda_1$ , and the second principal component captures the next largest variance, $\lambda_2$ , orthogonal to the first. If we decide to reduce the dimensionality from $p$ to $k$ by keeping only the first $k$ principal components, the proportion of the total variance retained is given by $\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^p \lambda_i}$ . Suppose for a particular dataset, the eigenvalues of the covariance matrix are $\{10.5, 5.2, 2.1, 0.8, 0.4 \}$ . If we aim to retain at least 90% of the total variance, what is the minimum number of principal components required?

Advanced

The total variance in the dataset is the sum of all eigenvalues: $\sum_{i=1}^p \lambda_i = 10.5 + 5.2 + 2.1 + 0.8 + 0.4 = 19.0$ .

We need to find the minimum $k$ such that $\frac{\sum_{i=1}^k \lambda_i}{\sum_{i=1}^p \lambda_i} \ge 0.90$ .

Let's check for different values of $k$ :

For $k=1$ : The variance retained is $\frac{10.5}{19.0} \approx 0.553$ , which is less than 0.90.

For $k=2$ : The variance retained is $\frac{10.5 + 5.2}{19.0} = \frac{15.7}{19.0} \approx 0.826$ , which is less than 0.90.

For $k=3$ : The variance retained is $\frac{10.5 + 5.2 + 2.1}{19.0} = \frac{17.8}{19.0} \approx 0.937$ , which is greater than or equal to 0.90.

Therefore, a minimum of 3 principal components are required to retain at least 90% of the total variance.

Consider a simple linear regression model where the predicted value $\hat{y}$ is given by $\hat{y} = b_0 + b_1 x$ . If we have two data points $(x_1, y_1) = (2, 5)$ and $(x_2, y_2) = (4, 9)$ , and the Ordinary Least Squares (OLS) method yields the regression line $\hat{y} = 1 + 2x$ , what is the predicted value $\hat{y}$ for $x = 3$ ?

Basic

The problem provides a simple linear regression equation $\hat{y} = b_0 + b_1 x$ and specific values for the intercept $b_0 = 1$ and the slope $b_1 = 2$ . We are asked to find the predicted value $\hat{y}$ for a given value of $x$ . \nWe substitute $x = 3$ into the regression equation: $\hat{y} = 1 + 2(3)$ $\hat{y} = 1 + 6$ $\hat{y} = 7$ \nTherefore, the predicted value $\hat{y}$ for $x = 3$ is 7. The data points $(2, 5)$ and $(4, 9)$ are consistent with this regression line: for $x=2$ , $\hat{y} = 1 + 2(2) = 5$ , and for $x=4$ , $\hat{y} = 1 + 2(4) = 9$ .

Final Answer: $7$

A binary classification model uses a logistic sigmoid function $\sigma(z) = \frac{1}{1 + e^{-z}}$ . The decision boundary is defined by $\sigma(z) = 0.5$ , where $z = w_1 x_1 + w_0$ . For a dataset with two features $x_1$ and $x_2$ , the model's linear component is $z = 2x_1 - x_2 + b$ . If the classification boundary passes through the point $(x_1, x_2) = (3, 1)$ , what is the value of the bias $b$ ?

Intermediate

The logistic sigmoid function is $\sigma(z) = \frac{1}{1 + e^{-z}}$ . The decision boundary is where $\sigma(z) = 0.5$ . Setting $\sigma(z) = 0.5$ gives $\frac{1}{1 + e^{-z}} = 0.5$ , which implies $1 = 0.5(1 + e^{-z})$ , so $2 = 1 + e^{-z}$ , and $e^{-z} = 1$ . This occurs when $-z = 0$ , or $z = 0$ . Therefore, the decision boundary is defined by $z = 0$ .

The linear component of the model is given by $z = w_1 x_1 + w_2 x_2 + b$ . In this problem, $z = 2x_1 - x_2 + b$ .

Since the decision boundary is $z = 0$ , we have $2x_1 - x_2 + b = 0$ .

The classification boundary passes through the point $(x_1, x_2) = (3, 1)$ . Substituting these values into the equation for the decision boundary:
$2(3) - 1 + b = 0$
$6 - 1 + b = 0$
$5 + b = 0$
$b = -5$

Final Answer: $b = -7$

Consider the following quadratic loss function: $L(w) = 3w^2 - 12w + 10$ . What is the value of $w$ that minimizes this loss function?

Basic

To find the minimum of the quadratic loss function $L(w) = 3w^2 - 12w + 10$ , we can find the derivative with respect to $w$ and set it to zero. The derivative of $L(w)$ with respect to $w$ is: $\frac{dL}{dw} = 6w - 12$ . Setting the derivative to zero to find the critical point: $6w - 12 = 0$ $6w = 12$ $w = \frac{12}{6}$ $w = 2$ . The second derivative is $\frac{d^2L}{dw^2} = 6$ , which is positive, confirming that $w=2$ corresponds to a minimum.

Consider a linear transformation represented by the matrix $M = \begin{bmatrix} 2 & k \\ 1 & 3 \end{bmatrix}$ . A vector $\mathbf{v} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ is transformed by $M$ into $\mathbf{v'} = M\mathbf{v} = \begin{bmatrix} 8 \\ 7 \end{bmatrix}$ . Determine the value of the unknown parameter $k$ .

Intermediate

The problem involves a matrix-vector multiplication. We are given the matrix $M = \begin{bmatrix} 2 & k \\ 1 & 3 \end{bmatrix}$ and the vectors $\mathbf{v} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ and $\mathbf{v'} = \begin{bmatrix} 8 \\ 7 \end{bmatrix}$ . The transformation is defined by $\mathbf{v'} = M\mathbf{v}$ . Let's perform the matrix-vector multiplication:

$\begin{bmatrix} 8 \\ 7 \end{bmatrix} = \begin{bmatrix} 2 & k \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

This expands to two equations:

1. $8 = (2)(1) + (k)(2)$
2. $7 = (1)(1) + (3)(2)$

Let's solve the first equation for $k$ :

$8 = 2 + 2k$

Subtract 2 from both sides:

$6 = 2k$

Divide by 2:

$k = 3$

Now, let's verify this with the second equation. The second equation is $7 = 1 + 6$ , which simplifies to $7 = 7$ . This confirms our value of $k = 3$ is consistent.

Therefore, the unknown parameter $k$ is 3.

A dataset consists of three independent observations $X_1, X_2, X_3$ drawn from a Poisson distribution with parameter $\lambda$ . The observed values are $x_1 = 3, x_2 = 5, x_3 = 4$ . We want to find the Maximum Likelihood Estimate (MLE) for $\lambda$ . The log-likelihood function for a Poisson distribution is given by $\ell(\lambda) = \sum_{i=1}^{n} \left( x_i \log(\lambda) - \log(x_i!) - \lambda \right)$ .
Part 1: Derive the expression for the derivative of the log-likelihood function with respect to $\lambda$ .
Part 2: Set the derivative to zero to find the MLE of $\lambda$ .

Intermediate

The log-likelihood function for $n=3$ observations is $\ell(\lambda) = (x_1 \log(\lambda) - \log(x_1!) - \lambda) + (x_2 \log(\lambda) - \log(x_2!) - \lambda) + (x_3 \log(\lambda) - \log(x_3!) - \lambda)$ .

Part 1: The derivative of the log-likelihood function with respect to $\lambda$ is:
$\frac{d\ell}{d\lambda} = \sum_{i=1}^{3} \left( \frac{x_i}{\lambda} - 1 \right) = \left( \frac{x_1}{\lambda} - 1 \right) + \left( \frac{x_2}{\lambda} - 1 \right) + \left( \frac{x_3}{\lambda} - 1 \right)$ .

Part 2: To find the MLE, we set the derivative to zero:
$\frac{x_1}{\lambda} - 1 + \frac{x_2}{\lambda} - 1 + \frac{x_3}{\lambda} - 1 = 0$
$\frac{x_1 + x_2 + x_3}{\lambda} - 3 = 0$
$\frac{x_1 + x_2 + x_3}{\lambda} = 3$
$\hat{\lambda} = \frac{x_1 + x_2 + x_3}{3}$ .
Substituting the observed values $x_1 = 3, x_2 = 5, x_3 = 4$ :
$\hat{\lambda} = \frac{3+5+4}{3} = \frac{12}{3} = 4$ .

Final Answer: $\hat{\lambda} = \frac{3+5+4}{3} = 4$

A vector $\mathbf{v}$ is transformed into a new vector $\mathbf{v}'$ by a linear transformation represented by the matrix $\mathbf{M}$ . We are given that $\mathbf{v} = \begin{pmatrix} 2 \ 3 \end{pmatrix}$ and $\mathbf{v}' = \begin{pmatrix} 7 \ 11 \end{pmatrix}$ . The transformation matrix is of the form $\mathbf{M} = \begin{pmatrix} a & 1 \ 4 & b \end{pmatrix}$ . Determine the value of $a$ and $b$ .

Intermediate

The transformation is defined by $\mathbf{v}' = \mathbf{M}\mathbf{v}$ . Substituting the given vectors and matrix form:

$\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} a & 1 \ 4 & b \end{pmatrix} \begin{pmatrix} 2 \ 3 \end{pmatrix}$

Performing the matrix multiplication:

$\begin{pmatrix} 7 \ 11 \end{pmatrix} = \begin{pmatrix} 2a + 3 \ 8 + 3b \end{pmatrix}$

Equating the corresponding components, we get a system of two linear equations:

1) $2a + 3 = 7$
2) $8 + 3b = 11$

Solving the first equation for $a$ :

$2a = 7 - 3$
$2a = 4$
$a = 2$

Solving the second equation for $b$ :

$3b = 11 - 8$
$3b = 3$
$b = 1$

Thus, $a = 2$ and $b = 1$ .

Final Answer: a = 2, b = 1

Consider two events, A and B, with the following probabilities: $P(A) = 0.4$ , $P(B) = 0.5$ , and $P(A \cap B) = 0.2$ . What is the conditional probability of A given B, denoted as $P(A|B)$ ?

Basic

The formula for conditional probability is $P(A|B) = \frac{P(A \cap B)}{P(B)}$ . Given $P(A \cap B) = 0.2$ and $P(B) = 0.5$ , we can substitute these values into the formula: $P(A|B) = \frac{0.2}{0.5} = 0.4$ . Therefore, the conditional probability of A given B is 0.4.

Consider a dataset with two features, $X_1$ and $X_2$ . The covariance matrix for this dataset is given by: $\Sigma = \begin{pmatrix} 10 & 5 \\ 5 & 20 \end{pmatrix}$ What is the variance of the first feature, $X_1$ ?

Basic

The covariance matrix $\Sigma$ for a dataset with two features $X_1$ and $X_2$ is structured as follows:
$\Sigma = \begin{pmatrix} Cov(X_1, X_1) & Cov(X_1, X_2) \\ Cov(X_2, X_1) & Cov(X_2, X_2) \end{pmatrix}$
By definition, the variance of a random variable is the covariance of the variable with itself. Therefore, $Var(X_1) = Cov(X_1, X_1)$ and $Var(X_2) = Cov(X_2, X_2)$ .

In the given covariance matrix:
$\Sigma = \begin{pmatrix} 10 & 5 \\ 5 & 20 \end{pmatrix}$
The element in the first row and first column represents $Cov(X_1, X_1)$ , which is the variance of $X_1$ .

Thus, the variance of $X_1$ is 10. The covariance between $X_1$ and $X_2$ is 5, and the variance of $X_2$ is 20.

Consider a simple logistic regression model where the decision boundary is defined by the linear equation $w_1x_1 + w_2x_2 + b = 0$ . If the weights are $w_1 = 2$ and $w_2 = 3$ , and the bias is $b = -7$ , what is the equation of the decision boundary?

Basic

The decision boundary in a logistic regression model is defined by the equation where the log-odds are zero. This corresponds to a linear combination of the input features and weights, plus the bias, being equal to zero. Given the weights $w_1 = 2$ and $w_2 = 3$ , and the bias $b = -7$ , the equation of the decision boundary is $w_1x_1 + w_2x_2 + b = 0$ . Substituting the given values, we get $2x_1 + 3x_2 + (-7) = 0$ , which simplifies to $2x_1 + 3x_2 - 7 = 0$ .

Final Answer: $2x_1 + 3x_2 - 7 = 0$

Consider a simple quadratic loss function $L(w) = (w^2 - 4w + 5)$ . We want to find the minimum of this loss function using gradient descent. The gradient of the loss function with respect to $w$ is given by $\frac{dL}{dw} = 2w - 4$ . If we start at an initial weight $w_0 = 5$ and use a learning rate $\alpha = 0.5$ , what will be the value of the weight $w_1$ after the first update step, i.e., $w_1 = w_0 - \alpha \frac{dL}{dw}(w_0)$ ?

Intermediate

The loss function is $L(w) = w^2 - 4w + 5$ . The gradient is $\frac{dL}{dw} = 2w - 4$ . We are given an initial weight $w_0 = 5$ and a learning rate $\alpha = 0.5$ . The gradient at $w_0$ is $\frac{dL}{dw}(w_0) = 2(5) - 4 = 10 - 4 = 6$ . The update rule for gradient descent is $w_1 = w_0 - \alpha \frac{dL}{dw}(w_0)$ . Substituting the values, we get $w_1 = 5 - 0.5 \times 6 = 5 - 3 = 2$ .

Final Answer: $4$

A linear transformation is defined by the matrix $M = \begin{pmatrix} 2 & 1 \\ 3 & 4 \end{pmatrix}$ . If an input vector is $v = \begin{pmatrix} x \\ 1 \end{pmatrix}$ , and the output vector after the transformation is $w = \begin{pmatrix} 5 \\ 7 \end{pmatrix}$ , what is the value of $x$ ?

Basic

The linear transformation is given by $w = Mv$ . Substituting the given matrices and vectors, we have:
$\begin{pmatrix} 3 \\ 7 \end{pmatrix} = \begin{pmatrix} 2 & 1 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} x \\ 1 \end{pmatrix}$
Performing the matrix multiplication on the right side gives:
$\begin{pmatrix} 3 \\ 7 \end{pmatrix} = \begin{pmatrix} 2x + 1(1) \\ 3x + 4(1) \end{pmatrix} = \begin{pmatrix} 2x + 1 \\ 3x + 4 \end{pmatrix}$
Equating the corresponding components of the vectors, we get two equations:
1. $3 = 2x + 1$
2. $7 = 3x + 4$

Solving the first equation for $x$ :
$3 - 1 = 2x$
$2 = 2x$
$x = \frac{2}{2}$
$x = 1$

Solving the second equation for $x$ :
$7 - 4 = 3x$
$3 = 3x$
$x = \frac{3}{3}$
$x = 1$

Both equations yield $x=1$ . Therefore, the value of $x$ is 1.

A dataset consists of $n=100$ independent and identically distributed (i.i.d.) observations $x_1, x_2, \dots, x_{100}$ . These observations are drawn from a Bernoulli distribution with parameter $p$ . The probability mass function (PMF) of a Bernoulli distribution is given by $P(X=x; p) = p^x (1-p)^{1-x}$ for $x \in \{0, 1\}$ . The likelihood function $L(p)$ for this dataset is the product of the PMFs of the individual observations. If we observe $k=60$ successes (i.e., $x_i = 1$ ) and $n-k=40$ failures (i.e., $x_i = 0$ ), what is the likelihood function $L(p)$ for $p$ ?

Basic

The likelihood function $L(p)$ is the product of the probabilities of observing each data point given the parameter $p$ . For i.i.d. Bernoulli trials, $L(p) = \prod_{i=1}^{n} P(X=x_i; p)$ . Given $n=100$ observations, with $k=60$ successes (where $x_i=1$ ) and $n-k=40$ failures (where $x_i=0$ ), the likelihood function can be written as:

$L(p) = \left( \prod_{i: x_i=1} p^1 (1-p)^{1-1} \right) \times \left( \prod_{i: x_i=0} p^0 (1-p)^{1-0} \right)$

$L(p) = \left( \prod_{i: x_i=1} p \right) \times \left( \prod_{i: x_i=0} (1-p) \right)$

Since there are $k=60$ terms of $p$ and $n-k=40$ terms of $(1-p)$ , the likelihood function simplifies to:

$L(p) = p^{60} (1-p)^{40}$

Final Answer: $p^{60} (1-p)^{40}$

Consider two vectors, $\mathbf{u} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ and $\mathbf{v} = \begin{bmatrix} 1 \\ -2 \end{bmatrix}$ . What is the vector representing $2\mathbf{u} - \mathbf{v}$ ?

Basic

To find the vector $2\mathbf{u} - \mathbf{v}$ , we first compute $2\mathbf{u}$ and then subtract $\mathbf{v}$ from the result.

$2\mathbf{u} = 2 \times \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 2 \times 3 \\ 2 \times 4 \end{bmatrix} = \begin{bmatrix} 6 \\ 8 \end{bmatrix}$ .

Now, we subtract $\mathbf{v}$ from $2\mathbf{u}$ :

$2\mathbf{u} - \mathbf{v} = \begin{bmatrix} 6 \\ 8 \end{bmatrix} - \begin{bmatrix} 1 \\ -2 \end{bmatrix} = \begin{bmatrix} 6 - 1 \\ 8 - (-2) \end{bmatrix} = \begin{bmatrix} 5 \\ 8 + 2 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix}$ .

Therefore, the resulting vector is $\begin{bmatrix} 5 \\ 10 \end{bmatrix}$ .

Consider a dataset of observations $\{x_i\}_{i=1}^n$ where each $x_i$ is a 2-dimensional vector. The sample covariance matrix for this dataset is given by:
$S = \begin{pmatrix} 4 & -2 \\ -2 & 1 \end{pmatrix}$
Principal Component Analysis (PCA) is applied to this dataset. The first principal component corresponds to the direction with the largest variance. Let the direction vector of the first principal component be $\mathbf{v}_1 = \begin{pmatrix} a \\ b \end{pmatrix}$ . The variance explained by this principal component is given by the corresponding eigenvalue $\lambda_1$ .
If $\mathbf{v}_1$ is a unit vector, what is the value of $a^2+b^2$ ?

Intermediate

The first principal component direction vector $\mathbf{v}_1$ is an eigenvector of the covariance matrix $S$ . The variance explained by this component is the corresponding eigenvalue $\lambda_1$ . The problem states that $\mathbf{v}_1 = \begin{pmatrix} a \\ b \end{pmatrix}$ is a unit vector. By definition, a unit vector has a magnitude of 1. The squared magnitude of a vector $\begin{pmatrix} a \\ b \end{pmatrix}$ is $a^2 + b^2$ . Therefore, if $\mathbf{v}_1$ is a unit vector, its squared magnitude must be 1, which means $a^2 + b^2 = 1$ . The specific values of $a$ and $b$ and the eigenvalues of $S$ are not needed to answer this question, as it directly asks for the property of a unit vector.

Consider the quadratic loss function $L(w) = w^2 - 4w + 3$ . Find the value of $w$ where the function reaches its minimum.

Basic

The given loss function is a quadratic function $L(w) = w^2 - 4w + 3$ . This represents a parabola opening upwards. The minimum value of a quadratic function of the form $aw^2 + bw + c$ occurs at $w = -\frac{b}{2a}$ . In this case, $a = 1$ and $b = -4$ . Therefore, the minimum occurs at $w = -\frac{-4}{2(1)} = \frac{4}{2} = 2$ . Alternatively, we can find the minimum by taking the derivative and setting it to zero: $L'(w) = 2w - 4$ . Setting $L'(w) = 0$ gives $2w - 4 = 0$ , which implies $2w = 4$ , so $w = 2$ .

A simple linear regression model has been fitted to a dataset. The model equation is given by $\hat{y} = ax + b$ . The regression line passes through the points $(2, 10)$ and $(5, 22)$ in the feature-target space. Determine the value of the intercept $b$ .

Intermediate

The problem states that the linear regression line $\hat{y} = ax + b$ passes through two points: $(x_1, y_1) = (2, 10)$ and $(x_2, y_2) = (5, 22)$ .

We can use these two points to form a system of linear equations to solve for the unknown parameters $a$ (the slope) and $b$ (the intercept).

Substituting the first point $(2, 10)$ into the equation $\hat{y} = ax + b$ gives:
$10 = a(2) + b$ (Equation 1)

Substituting the second point $(5, 22)$ into the equation $\hat{y} = ax + b$ gives:
$22 = a(5) + b$ (Equation 2)

To find the slope $a$ , we can subtract Equation 1 from Equation 2:
$(22 - 10) = (5a - 2a) + (b - b)$
$12 = 3a$
$a = \frac{12}{3} = 4$

Now that we have the value of $a$ , we can substitute it back into either Equation 1 or Equation 2 to solve for $b$ . Let's use Equation 1:
$10 = 4(2) + b$
$10 = 8 + b$
$b = 10 - 8$
$b = 2$

Let's verify with Equation 2:
$22 = 4(5) + b$
$22 = 20 + b$
$b = 22 - 20$
$b = 2$

Both equations yield $b = 2$ . Therefore, the intercept $b$ is 2.

Common algebraic traps include incorrectly calculating the slope or making sign errors when substituting values back into the equations. For instance, if one incorrectly calculated the slope as $\frac{3}{12}$ or made a sign error in $10 - 8$ , they might arrive at an incorrect intercept.

Consider a 2D dataset represented by the points $A = (1, 2)$ , $B = (4, 2)$ , and $C = (2, 5)$ . If this dataset undergoes a linear transformation represented by the matrix $M = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}$ , what are the coordinates of the transformed points $A'$ , $B'$ , and $C'$ ?

Basic

The transformation of a point $\mathbf{x}$ by a matrix $M$ is given by $M \mathbf{x}$ . We apply the matrix $M = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}$ to each point:

For point $A = (1, 2)$ , the transformed point $A'$ is:
$\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \end{pmatrix} = \begin{pmatrix} 2 \times 1 + 0 \times 2 \\ 0 \times 1 + 1 \times 2 \end{pmatrix} = \begin{pmatrix} 2 \\ 2 \end{pmatrix}$ So, $A' = (2, 2)$ .

For point $B = (4, 2)$ , the transformed point $B'$ is:
$\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 4 \\ 2 \end{pmatrix} = \begin{pmatrix} 2 \\ 2 \end{pmatrix}$ So, $B' = (8, 2)$ .

For point $C = (2, 5)$ , the transformed point $C'$ is:
$\begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix} \begin{pmatrix} 2 \\ 5 \end{pmatrix} = \begin{pmatrix} 2 \times 2 + 0 \times 5 \\ 0 \times 2 + 1 \times 5 \end{pmatrix} = \begin{pmatrix} 4 \\ 5 \end{pmatrix}$ So, $C' = (4, 5)$ .

Therefore, the transformed points are $A' = (2, 2)$ , $B' = (8, 2)$ , and $C' = (4, 5)$ .

Consider a discrete random variable $X$ with the following probability mass function (PMF): $P(X=x) = \frac{x}{10}$ for $x \in \{1, 2, 3, 4\}$ . What is the value of $P(X \le 2)$ ?

Basic

The probability mass function is given by $P(X=x) = \frac{x}{10}$ for $x \in \{1, 2, 3, 4\}$ . We need to find $P(X \le 2)$ . This is the sum of probabilities for $X=1$ and $X=2$ . $P(X \le 2) = P(X=1) + P(X=2)$ . $P(X=1) = \frac{1}{10}$ and $P(X=2) = \frac{2}{10}$ . Therefore, $P(X \le 2) = \frac{1}{10} + \frac{2}{10} = \frac{3}{10}$ .

Consider a dataset where the number of defective items produced by a machine per hour, $Y$ , follows a Poisson distribution with a rate parameter $\lambda = 3$ . The probability mass function (PMF) for a Poisson distribution is given by $P(Y=k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \dots$ . What is the value of $\lambda$ in this scenario?

Basic

The problem statement explicitly defines that the number of defective items, $Y$ , follows a Poisson distribution with a rate parameter $\lambda = 3$ . The rate parameter $\lambda$ is a direct characteristic of the Poisson distribution that determines its mean and variance. Therefore, the value of $\lambda$ is directly given as 3.

Consider a simple linear regression model where a dependent variable $y$ is predicted from an independent variable $x$ . The model is given by the equation $y = \beta_0 + \beta_1 x + \epsilon$ , where $\epsilon$ is the error term. Suppose we have estimated the regression line to be $\hat{y} = 3 + 2x$ . If we observe an independent variable value of $x=4$ , what is the predicted value $\hat{y}$ for the dependent variable?

Basic

The problem asks for the predicted value $\hat{y}$ given an independent variable value $x=4$ and a linear regression equation $\hat{y} = 3 + 2x$ . We substitute the value of $x$ into the equation:

$\hat{y} = 3 + 2(4)$
$\hat{y} = 3 + 8$
$\hat{y} = 11$

Therefore, the predicted value for the dependent variable is 11. Common errors might involve incorrectly multiplying 2 by 4 or incorrectly adding 3 to 8.

A centered feature matrix $\mathbf{X}$ has shape $N \times 22$ . What is the size and symmetry property of the sample covariance matrix $\mathbf{\Sigma} = \frac{1}{N-1}\mathbf{X}^T\mathbf{X}$ ?

Advanced

Since $\mathbf{X}$ has $22$ columns, $\mathbf{X}^T$ is $22 \times N$ and $\mathbf{X}^T\mathbf{X}$ is $22 \times 22$ . Furthermore, $(\mathbf{X}^T\mathbf{X})^T = \mathbf{X}^T(\mathbf{X}^T)^T = \mathbf{X}^T\mathbf{X}$ , proving it is always symmetric.

Consider a linear layer with a weight matrix $W$ and an input vector $\mathbf{x}$ . If $W = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}$ and $\mathbf{x} = \begin{pmatrix} 5 \\ 6 \end{pmatrix}$ , compute the output vector $\mathbf{y} = W\mathbf{x}$ .

Basic

The output vector $\mathbf{y}$ is computed by multiplying the weight matrix $W$ by the input vector $\mathbf{x}$ . This is a standard matrix-vector multiplication. \\ \\ $\mathbf{y} = W\mathbf{x} = \begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix} \begin{pmatrix} 5 \\ 6 \end{pmatrix}$ \\\ \\ To compute the first element of $\mathbf{y}$ , we take the dot product of the first row of $W$ with $\mathbf{x}$ : $(1)(5) + (2)(6) = 5 + 12 = 17$ .\\ \\ To compute the second element of $\mathbf{y}$ , we take the dot product of the second row of $W$ with $\mathbf{x}$ : $(3)(5) + (4)(6) = 15 + 24 = 39$ .\\ \\ Therefore, the output vector is $\mathbf{y} = \begin{pmatrix} 17 \\ 39 \end{pmatrix}$ .

Consider two vectors, $\mathbf{u} = \begin{pmatrix} 3 \\ -2 \end{pmatrix}$ and $\mathbf{v} = \begin{pmatrix} -1 \\ 5 \end{pmatrix}$ . What is the resultant vector $\mathbf{w} = \mathbf{u} + \mathbf{v}$ ?

Basic

To find the resultant vector $\mathbf{w} = \mathbf{u} + \mathbf{v}$ , we add the corresponding components of the two vectors. $\mathbf{w} = \begin{pmatrix} 3 \\ -2 \end{pmatrix} + \begin{pmatrix} -1 \\ 5 \end{pmatrix} = \begin{pmatrix} 3 + (-1) \\ -2 + 5 \end{pmatrix} = \begin{pmatrix} 2 \\ 3 \end{pmatrix}$ .

In training a language model token predictor, a binary token occurs $k = 15$ times across $n = 50$ independent trials. What is the Maximum Likelihood Estimate $\hat{p}_{\text{MLE}}$ of the token probability?

Advanced

For a Bernoulli likelihood $L(p) = p^k (1-p)^{n-k}$ , the log-likelihood is $\ln L = k\ln p + (n-k)\ln(1-p)$ . Setting derivative to zero: $\frac{k}{p} - \frac{n-k}{1-p} = 0 \implies \hat{p} = \frac{k}{n} = \frac{15}{50} = 0.300$ .

Consider the following quadratic loss function for a regression problem: $L(y, \hat{y}) = (y - \hat{y})^2$ . If the predicted value $\hat{y} = 5$ and the true value $y = 3$ , what is the value of the loss function?

Basic

The quadratic loss function is defined as $L(y, \hat{y}) = (y - \hat{y})^2$ . Given the true value $y = 3$ and the predicted value $\hat{y} = 5$ , we substitute these values into the loss function:

$L(3, 5) = (3 - 5)^2$

$L(3, 5) = (-2)^2$

$L(3, 5) = 4$

Therefore, the value of the loss function is 4.

Consider a 2D data point $\mathbf{v} = \begin{bmatrix} 3 \\ -4 \end{bmatrix}$ . What is the squared Euclidean norm (or squared magnitude) of this vector?

Basic

The squared Euclidean norm of a vector $\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \end{bmatrix}$ is given by $\|\mathbf{v}\|^2 = v_1^2 + v_2^2$ . For the given vector $\mathbf{v} = \begin{bmatrix} 3 \\ -4 \end{bmatrix}$ , the squared Euclidean norm is $(3)^2 + (-4)^2 = 9 + 16 = 25$ .

A vector $\mathbf{v}$ is represented in a 2D space. If the components of the vector are $v_x = -3$ and $v_y = 4$ , what is the Euclidean ( $L_2$ ) norm of this vector?

Basic

The Euclidean ( $L_2$ ) norm of a vector $\mathbf{v} = [v_x, v_y]^T$ is calculated as $\|\mathbf{v}\|_2 = \sqrt{v_x^2 + v_y^2}$ . For the given vector $\mathbf{v}$ with components $v_x = -3$ and $v_y = 4$ , the norm is: $\|\mathbf{v}\|_2 = \sqrt{(-3)^2 + (4)^2} = \sqrt{9 + 16} = \sqrt{25} = 5$ .

A machine learning diagnostic test has sensitivity $P(\text{Positive} | \text{Defect}) = 0.90$ . If the defect rate in production is $P(\text{Defect}) = 0.25$ , find the joint probability of a defect and a positive test result $P(\text{Defect} \cap \text{Positive})$ .

Intermediate

By the multiplication rule of conditional probability: $P(\text{Defect} \cap \text{Positive}) = P(\text{Defect}) \times P(\text{Positive} | \text{Defect}) = 0.25 \times 0.90 = 0.2250$ .

A 2D loss surface is defined as $L(w_1, w_2) = 2w_1^2 + 3w_2^2$ . Find the gradient vector $\nabla L$ at the point $(w_1, w_2) = (2, -1)$ .

Advanced

Compute partial derivatives: $\frac{\partial L}{\partial w_1} = 2(2)w_1 = 4(2) = 8$ ; $\frac{\partial L}{\partial w_2} = 2(3)w_2 = 6(-1) = -6$ . Therefore, $\nabla L = [8, -6]^T$ .

A medical diagnostic test for a rare disease is designed to be highly sensitive and specific. The disease affects $1$ in every $1000$ individuals in the population. The test has a sensitivity of $99$ \% (meaning it correctly identifies $99$ \% of individuals who have the disease) and a specificity of $95$ \% (meaning it correctly identifies $95$ \% of individuals who do not have the disease). If a randomly selected individual from the population tests positive, what is the probability that they actually have the disease?

Advanced

Let D be the event that an individual has the disease, and P be the event that the individual tests positive.
We are given the following probabilities:
- Prior probability of having the disease: $P(D) = \frac{1}{1000} = 0.001$
- Prior probability of not having the disease: $P(D^c) = 1 - P(D) = 1 - 0.001 = 0.999$
- Sensitivity (true positive rate): $P(P|D) = 0.99$
- Specificity (true negative rate): $P(P^c|D^c) = 0.95$
From specificity, we can derive the false positive rate: $P(P|D^c) = 1 - P(P^c|D^c) = 1 - 0.95 = 0.05$ .

We want to find the probability that an individual actually has the disease given a positive test result, which is $P(D|P)$ . We can use Bayes' Theorem:

$P(D|P) = \frac{P(P|D) P(D)}{P(P)}$

To find $P(P)$ , the overall probability of testing positive, we can use the law of total probability:

$P(P) = P(P|D) P(D) + P(P|D^c) P(D^c)$

Substitute the given values:

$P(P) = (0.99)(0.001) + (0.05)(0.999)$
$P(P) = 0.00099 + 0.04995$
$P(P) = 0.05094$

Now, substitute $P(P)$ back into Bayes' Theorem:

$P(D|P) = \frac{(0.99)(0.001)}{0.05094}$
$P(D|P) = \frac{0.00099}{0.05094}$
$P(D|P) \approx 0.0194346$

Therefore, the probability that a person who tests positive actually has the disease is approximately $0.0194$ , which is closest to $0.019$ .

Compute the Manhattan ( $L_1$ ) norm and Euclidean ( $L_2$ ) norm of the word vector $\mathbf{w} = [10, -15]^T$ .

Intermediate

$L_1$ norm is the sum of absolute values: $|10| + |-15| = 25$ . $L_2$ norm is the square root of sum of squares: $\sqrt{(10)^2 + (-15)^2} = \sqrt{100 + 225} = \sqrt{325}$ .

In a multiple linear regression model, the Sum of Squared Errors is $\text{SSE} = 20$ and the Total Sum of Squares is $\text{SST} = 100$ . What is the coefficient of determination $R^2$ ?

Intermediate

The coefficient of determination measures the proportion of variance explained: $R^2 = 1 - \frac{\text{SSE}}{\text{SST}} = 1 - \frac{20}{100} = 1 - 0.20 = 0.80$ .

In evaluating a binary classifier, the confusion matrix shows $TP = 70$ , $TN = 40$ , $FP = 10$ , and $FN = 10$ . Compute the overall classification accuracy.

Basic

Accuracy is the ratio of correct predictions to total samples: $\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN} = \frac{70 + 40}{130} = \frac{110}{130} = 84.6\%$ .

A fully-connected dense layer maps an input $\mathbf{x} \in \mathbb{R}^{50}$ to $\mathbf{y} \in \mathbb{R}^{25}$ using $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$ . How many total learnable parameters are in this layer?

Basic

The weight matrix $\mathbf{W}$ has shape $25 \times 50$ , giving $1250$ weights. The bias vector $\mathbf{b}$ has $25$ parameters. Total = $1250 + 25 = 1275$ .

A logistic regression model produces a class probability $p = P(y=1|\mathbf{x}) = 0.30$ . Compute the corresponding log-odds (logit) $\ln\left(\frac{p}{1-p}\right)$ .

Advanced

Odds are $\frac{p}{1-p} = \frac{0.30}{0.70} = 0.4286$ . Taking natural logarithm: $\ln(0.4286) = -0.847$ .

Evaluate the output of the standard ReLU activation function $f(z) = \max(0, z)$ for inputs $z_1 = -4$ and $z_2 = 6$ .

Intermediate

By definition of the Rectified Linear Unit: $\text{ReLU}(z) = z$ if $z \gt 0$ , and $0$ if $z \le 0$ . Thus $f(-4) = 0$ and $f(6) = 6$ .

AI-assisted, human-reviewed: Questions and walkthroughs are developed with AI assistance and reviewed by senior educators. Report an error here ↗

Generate an authentic assessment covering vector embeddings, neural feedforward calculations, gradient descent optimization, and classification metrics.

Questions

Difficulty

Source

Full official exam paper

Comprehensive 100-mark assessment covering the mathematical foundations required for modern Machine Learning, Deep Learning, and Data Science.

AI-assisted, human-reviewed: Questions and walkthroughs are developed with AI assistance and reviewed by senior educators. Report an error here ↗

↓

Downloads

Study Math for AI & Data Science offline. Download comprehensive cheat sheets on neural layer matrix operations, gradient updates, Bayes rules, and dimensionality reduction.

PDF · ready now

Math for AI & Data Science Handbook

Eight foundational chapters covering high-dimensional geometry, gradient descent, loss landscapes, and probabilistic modeling.

DOCX · on request

Native Word packet

Editable .docx problem sets with complete formulas and step-by-step mathematical proofs.

Official · Past Papers

Past Papers & Official Tests

Officially released exams and mark schemes from the examining body.

Mathematics for Machine Learning — Reference Library Academic reference on Linear Algebra, Calculus, and Probability for AI DeepLearning.AI Mathematics for Machine Learning Applied linear algebra, multivariate calculus, and statistics in Python

Worksheet Library

Curated collection of applied AI math drills, vector similarity exercises, and loss optimization challenges.

Loading worksheet library…

📋 Mistake Log

📊 Your Progress

Math for AI & Data Science

Vectors, Norms & High-Dimensional Embeddings

📊 Math for AI & Data Science — Exam Blueprint & Domain Weighting

Matrices, Linear Layers & Activation Functions

Loss Functions, Gradients & Optimization

Conditional Probability, Bayes' Rule & Classifiers

Probability Distributions & Maximum Likelihood (MLE)

Linear Regression & Ordinary Least Squares

Logistic Regression & Classification Boundaries

Dimensionality Reduction, Covariance & PCA

Practice set

Downloads

Math for AI & Data Science Handbook

Native Word packet

Past Papers & Official Tests

Worksheet Library

Privacy Policy

1. What's Stored Locally on Your Device

2. User Accounts, Cloud Sync & Third-Party Processors

3. ClipSAT AI Tutor & Practice Generation

4. Google Forms Integration & Google Drive Permissions

5. Advertising, Analytics & Cookie Consent

6. Saudi Personal Data Protection Law (PDPL) Compliance

7. Visitor Counter & Cookieless Analytics

8. Contact & Privacy Requests

Terms of Service

1. Core Content & Free-Tier Promise

2. ClipSAT Plus Subscriptions, Billing & 14-Day Refund Policy

3. ClipSAT AI Tutor Disclaimer

4. Acceptable Use of Content & Copyright

5. Educational Study Aid & No Guarantee of Scores

6. Contact & Support

Mock Exam

Math for AI & Data Scienceرياضيات الذكاء الاصطناعي وعلوم البيانات

Vectors, Norms & High-Dimensional Embeddingsالمتجهات والمسافات المعيارية والتمثيل الشعاعي (Embeddings)

📊 Math for AI & Data Science — Exam Blueprint & Domain Weighting

Matrices, Linear Layers & Activation Functionsالمصفوفات والطبقات الخطية في الشبكات العصبية ودوال التنشيط

Loss Functions, Gradients & Optimizationدوال الخسارة والانحدار التدريجي والتحسين الرياضي

Conditional Probability, Bayes' Rule & Classifiersالاحتمال الشرطي وقاعدة بايز ومصنفات الذكاء الاصطناعي

Probability Distributions & Maximum Likelihood (MLE)التوزيعات الاحتمالية وتقدير الاحتمال الأقصى (MLE)

Linear Regression & Ordinary Least Squaresالانحدار الخطي وطريقة المربعات الصغرى (OLS)

Logistic Regression & Classification Boundariesالانحدار اللوجستي والحدود الفاصلة للتصنيف

Dimensionality Reduction, Covariance & PCAتقليل الأبعاد ومصفوفة التباين المشترك وتحليل المكونات الرئيسية (PCA)

Practice setمجموعة تدريبية

Downloadsالتنزيلات

Math for AI & Data Science Handbookالرياضيات للذكاء الاصطناعي وتعلم الآلة (Math for AI) — Math for AI & Data Science Handbook

Native Word packet

Past Papers & Official Tests

Worksheet Libraryمكتبة أوراق العمل

Mock Exam

Math for AI & Data Science

Vectors, Norms & High-Dimensional Embeddings

Matrices, Linear Layers & Activation Functions

Loss Functions, Gradients & Optimization

Conditional Probability, Bayes' Rule & Classifiers

Probability Distributions & Maximum Likelihood (MLE)

Linear Regression & Ordinary Least Squares

Logistic Regression & Classification Boundaries

Dimensionality Reduction, Covariance & PCA

Practice set

Downloads

Math for AI & Data Science Handbook

Worksheet Library