Building MINST Neural Network only use Numpy

Building a neural network for MNIST from scratch without library like pytorch, keras, tensorflow, only use numpy. We will break down the mechanics step by step—from raw pixels to calculus-driven learning—so you have complete clarity before writing a single line of code. I’ll guide you through each piece with questions along the way.

To classify handwritten digits ($0$ through $9$), a standard feedforward neural network relies on three core conceptual pillars:

1. Architecture & Dimensions 📐: Turning a $28 \times 28$ grayscale image into vectors, and defining the matrix shapes for layers, weights ($W$), and biases ($b$).

2. Forward Propagation & Activation Functions ⚡: Computing the linear transformations ($Z = WX + b$) and applying non-linearities like ReLU for hidden layers and Softmax for multi-class probabilities.

3. Loss & Backpropagation 🔄: Measuring prediction error using Cross-Entropy Loss, applying the multivariable Chain Rule to compute gradients, and updating parameters via Gradient Descent.

Let’s set up the network architecture on paper! 📝

1. The Input Layer 🖼️

Each MNIST image is a grayscale image with a grid size of $28 \times 28$ pixels. Before feeding an image into a standard fully connected (dense) neural network, we need to flatten that 2D grid into a single 1D column vector, which we’ll call $X$.

To get started with our dimensions:

If we flatten a $28 \times 28$ pixel image into a single column vector, how many elements (features) will the vector $X$ have? What are its matrix dimensions $(n \times 1)$?

$28 \times 28 = 784$, so our input vector $X$ has 784 features with dimensions $784 \times 1$.

2. The Hidden Layer 🧠

Let’s design a simple 2-layer network (1 hidden layer + 1 output layer).

Suppose we choose our hidden layer to have $128$ neurons.

In the linear equation for this first layer:

$$Z^{[1]} = W^{[1]} X + b^{[1]}$$

• $X$ has dimensions $(784 \times 1)$.

We want the output of this layer, $Z^{[1]}$, to have a value for each of the $128$ neurons, giving it dimensions $(128 \times 1)$.

Using the rules of matrix multiplication ($(\text{rows} \times \text{cols}) \times (784 \times 1) = (128 \times 1)$), what must the dimensions of the weight matrix $W^{[1]}$ and the bias vector $b^{[1]}$ be?

For the weight matrix $W^{[1]}$, having dimensions of $128 \times 784$ allows $(128 \times 784) \times (784 \times 1)$ to produce a $(128 \times 1)$ output.

The bias vector $b^{[1]}$ is added directly to that result, so it matches the output shape: $128 \times 1$.

But how the matrix multiplication works mechanically and geometrically?

Instead of imagining an abstract block of math, look at it as 128 separate template matchers running in parallel across the image.

1. Break the Matrix Down into Rows

When you perform:

$\mathbf{y} = \mathbf{W} \mathbf{x} + \mathbf{b}$

• $\mathbf{x}$ is a vector of 784 numbers (the unrolled $28 \times 28$ image pixels).
• $\mathbf{W}$ is a matrix of size $128 \times 784$.
• $\mathbf{y}$ is a vector of 128 numbers (the hidden layer activations).

A matrix multiplication is just 128 dot products.

Each single neuron $i$ in that 128-neuron layer owns exactly one row of 784 weights in $\mathbf{W}$:

$y_i = \sum_{k=1}^{784} W_{i, k} x_k + b_i$

2. Every Neuron is an “Image” Itself

Because each row has 784 weights, you can reshape those 784 weights back into a $28 \times 28$ grid.

• Think of each of the 128 hidden neurons as carrying a $28 \times 28$ ghostly blueprint or filter.

• When that neuron does its dot product with the input image, it is computing a spatial cross-correlation:

  • Where the input pixel is bright and the neuron’s weight is positive, the activation shoots up.
  • Where the input pixel is bright but the weight is negative, it penalizes the activation.
  • Where both are zero, nothing happens.

3. What the 128 Neurons Actually “Look” For

In an untrained network, these 128 templates are static noise. But once trained via backpropagation on MNIST digits:

• Neuron #1 might have high weights along a diagonal slash (detecting the stroke in a 7 or /).
• Neuron #2 might develop positive weights arranged in a top loop (detecting the loop of an 8 or 9).
• Neuron #3 might activate on empty space in the center (ruling out 0s).

4. The Dimensionality Shift ($784 \to 128$)

You are not “losing” information arbitrarily; you are projecting the representation into a space of features:

• Input space (784 dimensions): “Is pixel (14, 12) dark?” (Raw sensory data, brittle to minor shifts).
• Hidden space (128 dimensions): “Does this image have a loop on top? A vertical spine? An open bottom?”(Semantic components).

By the time the next layer processes those 128 numbers to output the final 10 digits, it is no longer looking at raw pixels—it is just asking: “If it has a top circle (Neuron A = high) and a vertical stem (Neuron B = high), it’s probably a 9.”

3. Hidden Layer Activation ⚡

Before passing $Z^{[1]}$ to the next layer, we apply a non-linear activation function element-wise, commonly ReLU(Rectified Linear Unit):

$A^{[1]} = \text{ReLU}(Z^{[1]}) = \max(0, Z^{[1]})$

Because it operates element-by-element, the dimensions of the activated hidden layer $A^{[1]}$ remain $128 \times 1$.

4. The Output Layer 🎯

Now we need to project from the hidden layer ($128$ features) to our final predictions:

$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$

We want the final output to produce a prediction score for each possible digit in MNIST ($0, 1, 2, \dots, 9$).

How many output neurons do we need, and what must the dimensions of the second weight matrix $W^{[2]}$ and bias vector $b^{[2]}$ be?

We need 10 output neurons (one for each digit $0-9$). We now have our full network structure mapped out:

• Input $X$: $784 \times 1$
• Layer 1: $W^{[1]} \in \mathbb{R}^{128 \times 784}$, $b^{[1]} \in \mathbb{R}^{128 \times 1} \implies Z^{[1]}, A^{[1]} \in \mathbb{R}^{128 \times 1}$
• Layer 2: $W^{[2]} \in \mathbb{R}^{10 \times 128}$, $b^{[2]} \in \mathbb{R}^{10 \times 1} \implies Z^{[2]} \in \mathbb{R}^{10 \times 1}$

5. Output Activation: Softmax 📊

The vector $Z^{[2]}$ gives us 10 raw real-numbered scores (logits), but we want output probabilities that sum to $1$ for the classes $0$ through $9$.

We apply the Softmax function to compute the final activation $\hat{y} = A^{[2]}$:

$A^{[2]}_i = \frac{e^{Z^{[2]}_i}}{\sum_{j=1}^{10} e^{Z^{[2]}_j}}$

Now we have a probability vector $\hat{y}$ of size $10 \times 1$.

6. The Target Label 🎯

To calculate our loss (how wrong the prediction is), we need to compare our output probability vector $\hat{y}$ to the true label $y$.

If the true image is the digit $3$:

How do we represent this target digit $3$ as a $10 \times 1$ vector (known as a one-hot encoded vector) so it matches the shape of $\hat{y}$? What would that vector look like?

Because index $0$ represents the digit $0$, the digit $3$ actually sits at the 4th position (index 3):

$y = \begin{bmatrix} 0 \\ 0 \\ 0 \\ 1 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \end{bmatrix} \begin{matrix} \leftarrow \text{index 0 (digit 0)} \\ \leftarrow \text{index 1 (digit 1)} \\ \leftarrow \text{index 2 (digit 2)} \\ \leftarrow \text{index 3 (digit 3)} \\ \vdots \\ \leftarrow \text{index 9 (digit 9)} \end{matrix}$

With 0-based indexing, the $1$ goes at index 3, and all other entries are $0$.

7. Measuring Error: Categorical Cross-Entropy Loss 📉

Now that we have the prediction vector $\hat{y}$ and the true one-hot vector $y$, we compute the loss for a single sample using Categorical Cross-Entropy:

$L(y, \hat{y}) = -\sum_{i=0}^{9} y_i \ln(\hat{y}_i)$

Since $y$ has a $1$ only at the true class index (index 3 here) and $0$ everywhere else, what does this entire summation simplify to?

The loss simplifies down to:

$L = -\ln(\hat{y}_3)$

In general, for any true label class $k$, the cross-entropy loss simply measures the negative natural log of our predicted probability for that correct class: $L = -\ln(\hat{y}_k)$. If $\hat{y}_k \approx 1$, $L \approx 0$; if $\hat{y}_k \to 0$, $L \to \infty$.

8. Backpropagation: The Output Gradient 🔄

To update our weights using gradient descent, we need to calculate how changing each parameter affects the loss $L$ using the multivariable Chain Rule ⛓️.

We start at the very end and work backwards. The first key quantity is the derivative of the loss with respect to the output logits $Z^{[2]}$, which we denote as $dZ^{[2]}$:

$dZ^{[2]} = \frac{\partial L}{\partial Z^{[2]}}$

When you combine the derivative of the Categorical Cross-Entropy loss with the Softmax activation function using the chain rule ($\frac{\partial L}{\partial Z^{[2]}_i} = \sum_j \frac{\partial L}{\partial \hat{y}_j} \frac{\partial \hat{y}_j}{\partial Z^{[2]}_i}$), an algebraic cancellation occurs.

In terms of our prediction vector $\hat{y}$ and the true label vector $y$, do you know (or would you like to derive) the formula for $dZ^{[2]}$?

When you combine Categorical Cross-Entropy with Softmax, the calculus cancels out cleanly to:

$dZ^{[2]} = \hat{y} - y$

This is the error vector 🎯: it is simply our predicted probability vector minus the true one-hot vector.

• Its dimensions are $10 \times 1$, exactly matching $Z^{[2]}$.

9. Gradients for Layer 2 ($dW^{[2]}$ and $db^{[2]}$) 📐

Now we use $dZ^{[2]}$ to find how the loss changes with respect to the weights $W^{[2]}$ and biases $b^{[2]}$.

Recall the forward relationship:

$Z^{[2]} = W^{[2]} A^{[1]} + b^{[2]}$

By the chain rule:

• The bias gradient is simply: $db^{[2]} = dZ^{[2]}$ (shape: $10 \times 1$).
• For the weight gradient $dW^{[2]}$, we combine $dZ^{[2]}$ (shape $10 \times 1$) and the hidden activation $A^{[1]}$ (shape $128 \times 1$).

We know $dW^{[2]}$ must match the shape of $W^{[2]}$, which is $10 \times 128$.

How should we multiply $dZ^{[2]}$ and $A^{[1]}$ (using the transpose $(A^{[1]})^T$) so that the resulting matrix has dimensions $10 \times 128$?

To obtain a $10 \times 128$ matrix from a $10 \times 1$ vector and a $128 \times 1$ vector, we multiply $dZ^{[2]}$ by the transpose of $A^{[1]}$:

$dW^{[2]} = dZ^{[2]} (A^{[1]})^T$

• Matrix dimensions: $(10 \times 1) \times (1 \times 128) = 10 \times 128$.