Building MINST Neural Network only use Numpy

Building a neural network for MNIST from scratch is a fantastic pen-and-paper exercise. We will break down the mechanics step by step—from raw pixels to calculus-driven learning—so you have complete clarity before writing a single line of code. I’ll guide you through each piece with questions along the way.

To classify handwritten digits ($0$ through $9$), a standard feedforward neural network relies on three core conceptual pillars:

1. Architecture & Dimensions 📐: Turning a $28 \times 28$ grayscale image into vectors, and defining the matrix shapes for layers, weights ($W$), and biases ($b$).

2. Forward Propagation & Activation Functions ⚡: Computing the linear transformations ($Z = WX + b$) and applying non-linearities like ReLU for hidden layers and Softmax for multi-class probabilities.

3. Loss & Backpropagation 🔄: Measuring prediction error using Cross-Entropy Loss, applying the multivariable Chain Rule to compute gradients, and updating parameters via Gradient Descent.

Let’s set up the network architecture on paper! 📝

1. The Input Layer 🖼️

Each MNIST image is a grayscale image with a grid size of $28 \times 28$ pixels. Before feeding an image into a standard fully connected (dense) neural network, we need to flatten that 2D grid into a single 1D column vector, which we’ll call $X$.

To get started with our dimensions:

[Q]

If we flatten a $28 \times 28$ pixel image into a single column vector, how many elements (features) will the vector $X$ have? What are its matrix dimensions $(n \times 1)$?

[A]

$28 \times 28 = 784$, so our input vector $X$ has 784 features with dimensions $784 \times 1$.

2. The Hidden Layer 🧠

Let’s design a simple 2-layer network (1 hidden layer + 1 output layer).

Suppose we choose our hidden layer to have $128$ neurons.

In the linear equation for this first layer:

$Z^{[1]} = W^{[1]} X + b^{[1]}$

• $X$ has dimensions $(784 \times 1)$.

We want the output of this layer, $Z^{[1]}$, to have a value for each of the $128$ neurons, giving it dimensions $(128 \times 1)$.

[Q]

Using the rules of matrix multiplication ($(\text{rows} \times \text{cols}) \times (784 \times 1) = (128 \times 1)$), what must the dimensions of the weight matrix $W^{[1]}$ and the bias vector $b^{[1]}$ be?

[A]

For the weight matrix $W^{[1]}$, having dimensions of $128 \times 784$ allows $(128 \times 784) \times (784 \times 1)$ to produce a $(128 \times 1)$ output.

The bias vector $b^{[1]}$ is added directly to that result, so it matches the output shape: $128 \times 1$.

3. Hidden Layer Activation ⚡

Before passing $Z^{[1]}$ to the next layer, we apply a non-linear activation function element-wise, commonly ReLU(Rectified Linear Unit):

$A^{[1]} = \text{ReLU}(Z^{[1]}) = \max(0, Z^{[1]})$

Because it operates element-by-element, the dimensions of the activated hidden layer $A^{[1]}$ remain $128 \times 1$.

4. The Output Layer 🎯

Now we need to project from the hidden layer ($128$ features) to our final predictions:

$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$

We want the final output to produce a prediction score for each possible digit in MNIST ($0, 1, 2, \dots, 9$).

[Q]

How many output neurons do we need, and what must the dimensions of the second weight matrix $W^{[2]}$ and bias vector $b^{[2]}$ be?

[A]

We need 10 output neurons (one for each digit $0-9$). We now have our full network structure mapped out:

• Input $X$: $784 \times 1$
• Layer 1: $W^{[1]} \in \mathbb{R}^{128 \times 784}$, $b^{[1]} \in \mathbb{R}^{128 \times 1} \implies Z^{[1]}, A^{[1]} \in \mathbb{R}^{128 \times 1}$
• Layer 2: $W^{[2]} \in \mathbb{R}^{10 \times 128}$, $b^{[2]} \in \mathbb{R}^{10 \times 1} \implies Z^{[2]} \in \mathbb{R}^{10 \times 1}$

5. Output Activation: Softmax 📊

The vector $Z^{[2]}$ gives us 10 raw real-numbered scores (logits), but we want output probabilities that sum to $1$ for the classes $0$ through $9$.

We apply the Softmax function to compute the final activation $\hat{y} = A^{[2]}$:

$A^{[2]}_i = \frac{e^{Z^{[2]}_i}}{\sum_{j=1}^{10} e^{Z^{[2]}_j}}$

Now we have a probability vector $\hat{y}$ of size $10 \times 1$.

6. The Target Label 🎯

To calculate our loss (how wrong the prediction is), we need to compare our output probability vector $\hat{y}$ to the true label $y$.

If the true image is the digit $3$:

[Q]

How do we represent this target digit $3$ as a $10 \times 1$ vector (known as a one-hot encoded vector) so it matches the shape of $\hat{y}$? What would that vector look like?

[A]

Because index $0$ represents the digit $0$, the digit $3$ actually sits at the 4th position (index 3):

$y = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix} \begin{matrix} \leftarrow \text{index 0 (digit 0)} \\ \leftarrow \text{index 1 (digit 1)} \\ \leftarrow \text{index 2 (digit 2)} \\ \leftarrow \text{index 3 (digit 3)} \\ \vdots \\ \leftarrow \text{index 9 (digit 9)} \end{matrix}$

With 0-based indexing, the $1$ goes at index 3, and all other entries are $0$.

7. Measuring Error: Categorical Cross-Entropy Loss 📉

Now that we have the prediction vector $\hat{y}$ and the true one-hot vector $y$, we compute the loss for a single sample using Categorical Cross-Entropy:

$L(y, \hat{y}) = -\sum_{i=0}^{9} y_i \ln(\hat{y}_i)$

[Q]

Since $y$ has a $1$ only at the true class index (index 3 here) and $0$ everywhere else, what does this entire summation simplify to?

[A]

The loss simplifies down to:

$L = -\ln(\hat{y}_3)$

In general, for any true label class $k$, the cross-entropy loss simply measures the negative natural log of our predicted probability for that correct class: $L = -\ln(\hat{y}_k)$. If $\hat{y}_k \approx 1$, $L \approx 0$; if $\hat{y}_k \to 0$, $L \to \infty$.

8. Backpropagation: The Output Gradient 🔄

To update our weights using gradient descent, we need to calculate how changing each parameter affects the loss $L$ using the multivariable Chain Rule ⛓️.

We start at the very end and work backwards. The first key quantity is the derivative of the loss with respect to the output logits $Z^{[2]}$, which we denote as $dZ^{[2]}$:

$dZ^{[2]} = \frac{\partial L}{\partial Z^{[2]}}$

When you combine the derivative of the Categorical Cross-Entropy loss with the Softmax activation function using the chain rule ($\frac{\partial L}{\partial Z^{[2]}_i} = \sum_j \frac{\partial L}{\partial \hat{y}_j} \frac{\partial \hat{y}_j}{\partial Z^{[2]}_i}$), an algebraic cancellation occurs.

[Q]

In terms of our prediction vector $\hat{y}$ and the true label vector $y$, do you know (or would you like to derive) the formula for $dZ^{[2]}$?

[A]

When you combine Categorical Cross-Entropy with Softmax, the calculus cancels out cleanly to:

$dZ^{[2]} = \hat{y} - y$

This is the error vector 🎯: it is simply our predicted probability vector minus the true one-hot vector.

• Its dimensions are $10 \times 1$, exactly matching $Z^{[2]}$.

9. Gradients for Layer 2 ($dW^{[2]}$ and $db^{[2]}$) 📐

Now we use $dZ^{[2]}$ to find how the loss changes with respect to the weights $W^{[2]}$ and biases $b^{[2]}$.

Recall the forward relationship:

$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$

By the chain rule:

• The bias gradient is simply: $db^{[2]} = dZ^{[2]}$ (shape: $10 \times 1$).
• For the weight gradient $dW^{[2]}$, we combine $dZ^{[2]}$ (shape $10 \times 1$) and the hidden activation $A^{[1]}$ (shape $128 \times 1$).

We know $dW^{[2]}$ must match the shape of $W^{[2]}$, which is $10 \times 128$.

[Q]

How should we multiply $dZ^{[2]}$ and $A^{[1]}$ (using the transpose $(A^{[1]})^T$) so that the resulting matrix has dimensions $10 \times 128$?

[A]

To obtain a $10 \times 128$ matrix from a $10 \times 1$ vector and a $128 \times 1$ vector, we multiply $dZ^{[2]}$ by the transpose of $A^{[1]}$:

$dW^{[2]} = dZ^{[2]} (A^{[1]})^T$

• Matrix dimensions: $(10 \times 1) \times (1 \times 128) = 10 \times 128$.