Theory

Understand backpropagation down to the metal: micro-net

4 min read
615 words
# AI # deep-learning

import Info from ’@/components/widgets/Info.astro’;

Step 0: The Micro-Network Blueprint

To understand backpropagation down to the metal, strip away high-dimensional matrices and trace a single training example through a minimal network: 2 inputs →\to 2 hidden neurons with ReLU →\to 2 output logits with Softmax →\to Cross-Entropy Loss.

Plaintext

[x₁] ──W₁₁──► (z₁ → a₁) ──V₁₁──► (ẑ₁ → p₁) ──┐
  ╳            ╳                  ╳            ├──► Loss L
[x₂] ──W₂₂──► (z₂ → a₂) ──V₂₂──► (ẑ₂ → p₂) ──┘

Forward Variables & Notation

  • Inputs: x=[x1x2]x = \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}, Target: y=[y1y2]y = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} (one-hot, where y1=1,y2=0y_1 = 1, y_2 = 0).

  • Hidden Layer weights (WW) and biases (bb):

    z1=W11x1+W12x2+b1  ⟹  a1=max⁡(0,z1)z_1 = W_{11}x_1 + W_{12}x_2 + b_1 \implies a_1 = \max(0, z_1)

    z2=W21x1+W22x2+b2  ⟹  a2=max⁡(0,z2)z_2 = W_{21}x_1 + W_{22}x_2 + b_2 \implies a_2 = \max(0, z_2)

  • Output Layer weights (VV) and biases (cc):

    z^1=V11a1+V12a2+c1\hat{z}_1 = V_{11}a_1 + V_{12}a_2 + c_1

    z^2=V21a1+V22a2+c2\hat{z}_2 = V_{21}a_1 + V_{22}a_2 + c_2

  • Softmax Probabilities:

    p1=ez^1ez^1+ez^2,p2=ez^2ez^1+ez^2p_1 = \frac{e^{\hat{z}_1}}{e^{\hat{z}_1} + e^{\hat{z}_2}}, \quad p_2 = \frac{e^{\hat{z}_2}}{e^{\hat{z}_1} + e^{\hat{z}_2}}

  • Categorical Cross-Entropy Loss:

    L=−(y1ln⁡p1+y2ln⁡p2)L = -(y_1 \ln p_1 + y_2 \ln p_2)

Step 1: Deriving the Output Error ∂L∂z^i\frac{\partial L}{\partial \hat{z}_i}

The biggest algebraic mystery in deep learning is why ∂L∂z^=p−y\frac{\partial L}{\partial \hat{z}} = p - y. Here is the exact scalar proof using the quotient rule.

Case A: Derivative of Softmax pip_i with respect to its own logit z^i\hat{z}_i

Let S=ez^1+ez^2S = e^{\hat{z}_1} + e^{\hat{z}_2}. Then p1=ez^1Sp_1 = \frac{e^{\hat{z}_1}}{S}.

Using the quotient rule (uv)′=u′v−uv′v2\left(\frac{u}{v}\right)' = \frac{u'v - uv'}{v^2}:

∂p1∂z^1=∂(ez^1)∂z^1⋅S−ez^1⋅∂S∂z^1S2=ez^1⋅S−ez^1⋅ez^1S2\frac{\partial p_1}{\partial \hat{z}_1} = \frac{\frac{\partial (e^{\hat{z}_1})}{\partial \hat{z}_1} \cdot S - e^{\hat{z}_1} \cdot \frac{\partial S}{\partial \hat{z}_1}}{S^2} = \frac{e^{\hat{z}_1} \cdot S - e^{\hat{z}_1} \cdot e^{\hat{z}_1}}{S^2}

Split the fraction:

∂p1∂z^1=ez^1S(S−ez^1S)=ez^1S(1−ez^1S)=p1(1−p1)\frac{\partial p_1}{\partial \hat{z}_1} = \frac{e^{\hat{z}_1}}{S} \left(\frac{S - e^{\hat{z}_1}}{S}\right) = \frac{e^{\hat{z}_1}}{S} \left(1 - \frac{e^{\hat{z}_1}}{S}\right) = p_1(1 - p_1)

Case B: Derivative of Softmax pjp_j with respect to a different logit z^i\hat{z}_i (j≠ij \neq i)

Look at p2=ez^2Sp_2 = \frac{e^{\hat{z}_2}}{S} differentiated with respect to z^1\hat{z}_1:

∂p2∂z^1=0⋅S−ez^2⋅∂S∂z^1S2=−ez^2⋅ez^1S2=−(ez^1S)(ez^2S)=−p1p2\frac{\partial p_2}{\partial \hat{z}_1} = \frac{0 \cdot S - e^{\hat{z}_2} \cdot \frac{\partial S}{\partial \hat{z}_1}}{S^2} = \frac{-e^{\hat{z}_2} \cdot e^{\hat{z}_1}}{S^2} = -\left(\frac{e^{\hat{z}_1}}{S}\right)\left(\frac{e^{\hat{z}_2}}{S}\right) = -p_1 p_2

Case C: Applying the Chain Rule to Cross-Entropy

Since z^1\hat{z}_1 alters both p1p_1 and p2p_2, and both enter L=−y1ln⁡p1−y2ln⁡p2L = -y_1 \ln p_1 - y_2 \ln p_2:

∂L∂z^1=∂L∂p1∂p1∂z^1+∂L∂p2∂p2∂z^1\frac{\partial L}{\partial \hat{z}_1} = \frac{\partial L}{\partial p_1}\frac{\partial p_1}{\partial \hat{z}_1} + \frac{\partial L}{\partial p_2}\frac{\partial p_2}{\partial \hat{z}_1}

Substitute the loss derivatives ∂L∂p1=−y1p1\frac{\partial L}{\partial p_1} = -\frac{y_1}{p_1} and ∂L∂p2=−y2p2\frac{\partial L}{\partial p_2} = -\frac{y_2}{p_2}:

∂L∂z^1=(−y1p1)⋅[p1(1−p1)]+(−y2p2)⋅[−p1p2]\frac{\partial L}{\partial \hat{z}_1} = \left(-\frac{y_1}{p_1}\right) \cdot [p_1(1 - p_1)] + \left(-\frac{y_2}{p_2}\right) \cdot [-p_1 p_2]

Cancel terms:

∂L∂z^1=−y1(1−p1)+y2p1=−y1+y1p1+y2p1=p1(y1+y2)−y1\frac{\partial L}{\partial \hat{z}_1} = -y_1(1 - p_1) + y_2 p_1 = -y_1 + y_1 p_1 + y_2 p_1 = p_1(y_1 + y_2) - y_1

Because probabilities in one-hot vectors sum to 11 (y1+y2=1y_1 + y_2 = 1):

∂L∂z^1=p1(1)−y1=p1−y1\frac{\partial L}{\partial \hat{z}_1} = p_1(1) - y_1 = p_1 - y_1

Define this scalar error signal as:

δ1[2]=p1−y1,δ2[2]=p2−y2\delta_1^{[2]} = p_1 - y_1, \quad \delta_2^{[2]} = p_2 - y_2

Step 2: Output Layer Parameter Gradients

Now calculate how V11V_{11} and c1c_1 change the loss.

Weight V11V_{11}

V11V_{11} only influences LL through z^1=V11a1+V12a2+c1\hat{z}_1 = V_{11}a_1 + V_{12}a_2 + c_1.

∂L∂V11=∂L∂z^1⋅∂z^1∂V11\frac{\partial L}{\partial V_{11}} = \frac{\partial L}{\partial \hat{z}_1} \cdot \frac{\partial \hat{z}_1}{\partial V_{11}}

Since ∂z^1∂V11=a1\frac{\partial \hat{z}_1}{\partial V_{11}} = a_1:

∂L∂V11=δ1[2]⋅a1\frac{\partial L}{\partial V_{11}} = \delta_1^{[2]} \cdot a_1

Bias c1c_1

Since ∂z^1∂c1=1\frac{\partial \hat{z}_1}{\partial c_1} = 1:

∂L∂c1=∂L∂z^1⋅∂z^1∂c1=δ1[2]⋅1=δ1[2]\frac{\partial L}{\partial c_1} = \frac{\partial L}{\partial \hat{z}_1} \cdot \frac{\partial \hat{z}_1}{\partial c_1} = \delta_1^{[2]} \cdot 1 = \delta_1^{[2]}

Following the exact same path for all other weights in Layer 2:

∂L∂V12=δ1[2]a2,∂L∂V21=δ2[2]a1,∂L∂V22=δ2[2]a2,∂L∂c2=δ2[2]\frac{\partial L}{\partial V_{12}} = \delta_1^{[2]} a_2, \quad \frac{\partial L}{\partial V_{21}} = \delta_2^{[2]} a_1, \quad \frac{\partial L}{\partial V_{22}} = \delta_2^{[2]} a_2, \quad \frac{\partial L}{\partial c_2} = \delta_2^{[2]}

Step 3: Propagating Back to the Hidden Activations

How sensitive is LL to hidden activation a1a_1?

Notice that a1a_1 connects to both output neurons:

  • Path 1: a1→z^1→La_1 \to \hat{z}_1 \to L via weight V11V_{11}

  • Path 2: a1→z^2→La_1 \to \hat{z}_2 \to L via weight V21V_{21}

By the multivariable chain rule:

∂L∂a1=∂L∂z^1∂z^1∂a1+∂L∂z^2∂z^2∂a1=δ1[2]V11+δ2[2]V21\frac{\partial L}{\partial a_1} = \frac{\partial L}{\partial \hat{z}_1}\frac{\partial \hat{z}_1}{\partial a_1} + \frac{\partial L}{\partial \hat{z}_2}\frac{\partial \hat{z}_2}{\partial a_1} = \delta_1^{[2]} V_{11} + \delta_2^{[2]} V_{21}

Notice the index orientation: δ1[2]\delta_1^{[2]} multiplies V11V_{11} and δ2[2]\delta_2^{[2]} multiplies V21V_{21}. This matches the column of VV:

[∂L∂a1∂L∂a2]=[V11V21V12V22][δ1[2]δ2[2]]=VTδ[2]\begin{bmatrix} \frac{\partial L}{\partial a_1} \\ \frac{\partial L}{\partial a_2} \end{bmatrix} = \begin{bmatrix} V_{11} & V_{21} \\ V_{12} & V_{22} \end{bmatrix} \begin{bmatrix} \delta_1^{[2]} \\ \delta_2^{[2]} \end{bmatrix} = V^T \delta^{[2]}

The matrix transpose is an explicit outcome of collecting multi-path downstream connections.

Step 4: Pushing Through the Non-Linearity (ReLU)

To compute δ1[1]=∂L∂z1\delta_1^{[1]} = \frac{\partial L}{\partial z_1}, trace through the activation function a1=max⁡(0,z1)a_1 = \max(0, z_1):

∂L∂z1=∂L∂a1⋅∂a1∂z1\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1}

The derivative of ReLU is:

∂a1∂z1={1if z1>00if z1≤0\frac{\partial a_1}{\partial z_1} = \begin{cases} 1 & \text{if } z_1 > 0 \\ 0 & \text{if } z_1 \le 0 \end{cases}

Therefore:

δ1[1]=(δ1[2]V11+δ2[2]V21)⋅I(z1>0)\delta_1^{[1]} = (\delta_1^{[2]} V_{11} + \delta_2^{[2]} V_{21}) \cdot \mathbb{I}(z_1 > 0)

δ2[1]=(δ1[2]V12+δ2[2]V22)⋅I(z2>0)\delta_2^{[1]} = (\delta_1^{[2]} V_{12} + \delta_2^{[2]} V_{22}) \cdot \mathbb{I}(z_2 > 0)

Step 5: Input Layer Parameter Gradients

With δ1[1]\delta_1^{[1]} and δ2[1]\delta_2^{[1]} established, the first layer parameters mirror Step 2:

z1=W11x1+W12x2+b1z_1 = W_{11}x_1 + W_{12}x_2 + b_1

  • Weight W11W_{11}:

    ∂L∂W11=∂L∂z1⋅∂z1∂W11=δ1[1]⋅x1\frac{\partial L}{\partial W_{11}} = \frac{\partial L}{\partial z_1} \cdot \frac{\partial z_1}{\partial W_{11}} = \delta_1^{[1]} \cdot x_1

  • Weight W12W_{12}:

    ∂L∂W12=∂L∂z1⋅∂z1∂W12=δ1[1]⋅x2\frac{\partial L}{\partial W_{12}} = \frac{\partial L}{\partial z_1} \cdot \frac{\partial z_1}{\partial W_{12}} = \delta_1^{[1]} \cdot x_2

  • Bias b1b_1:

    ∂L∂b1=∂L∂z1⋅∂z1∂b1=δ1[1]⋅1=δ1[1]\frac{\partial L}{\partial b_1} = \frac{\partial L}{\partial z_1} \cdot \frac{\partial z_1}{\partial b_1} = \delta_1^{[1]} \cdot 1 = \delta_1^{[1]}

Likewise for neuron 2:

∂L∂W21=δ2[1]x1,∂L∂W22=δ2[1]x2,∂L∂b2=δ2[1]\frac{\partial L}{\partial W_{21}} = \delta_2^{[1]} x_1, \quad \frac{\partial L}{\partial W_{22}} = \delta_2^{[1]} x_2, \quad \frac{\partial L}{\partial b_2} = \delta_2^{[1]}

Hands-On Numerical Run (Trace with Numbers)

Let’s plug in raw numbers for one full backward step.

Given State:

  • Inputs: x1=2.0,x2=1.0x_1 = 2.0, \quad x_2 = 1.0

  • Ground Truth: y1=1,y2=0y_1 = 1, \quad y_2 = 0

  • Hidden Activations (assume positive zz, so ReLU′=1\text{ReLU}' = 1): a1=0.8,a2=0.4a_1 = 0.8, \quad a_2 = 0.4

  • Output Weights: V=[0.5−0.2−0.10.4]V = \begin{bmatrix} 0.5 & -0.2 \\ -0.1 & 0.4 \end{bmatrix}

  • Model Softmax Output: p1=0.3,p2=0.7p_1 = 0.3, \quad p_2 = 0.7

Manual Derivative Calculation:

  1. Output Errors:

    δ1[2]=p1−y1=0.3−1.0=−0.7\delta_1^{[2]} = p_1 - y_1 = 0.3 - 1.0 = -0.7

    δ2[2]=p2−y2=0.7−0.0=+0.7\delta_2^{[2]} = p_2 - y_2 = 0.7 - 0.0 = +0.7

  2. Output Weight Gradients:

    ∂L∂V11=δ1[2]⋅a1=(−0.7)(0.8)=−0.56\frac{\partial L}{\partial V_{11}} = \delta_1^{[2]} \cdot a_1 = (-0.7)(0.8) = -0.56

    ∂L∂V12=δ1[2]⋅a2=(−0.7)(0.4)=−0.28\frac{\partial L}{\partial V_{12}} = \delta_1^{[2]} \cdot a_2 = (-0.7)(0.4) = -0.28

    ∂L∂V21=δ2[2]⋅a1=(0.7)(0.8)=+0.56\frac{\partial L}{\partial V_{21}} = \delta_2^{[2]} \cdot a_1 = (0.7)(0.8) = +0.56

    ∂L∂V22=δ2[2]⋅a2=(0.7)(0.4)=+0.28\frac{\partial L}{\partial V_{22}} = \delta_2^{[2]} \cdot a_2 = (0.7)(0.4) = +0.28

  3. Propagating Back to Hidden Units:

    δ1[1]=(δ1[2]V11+δ2[2]V21)⋅1=(−0.7)(0.5)+(0.7)(−0.1)=−0.35−0.07=−0.42\delta_1^{[1]} = (\delta_1^{[2]} V_{11} + \delta_2^{[2]} V_{21}) \cdot 1 = (-0.7)(0.5) + (0.7)(-0.1) = -0.35 - 0.07 = -0.42

    δ2[1]=(δ1[2]V12+δ2[2]V22)⋅1=(−0.7)(−0.2)+(0.7)(0.4)=0.14+0.28=+0.42\delta_2^{[1]} = (\delta_1^{[2]} V_{12} + \delta_2^{[2]} V_{22}) \cdot 1 = (-0.7)(-0.2) + (0.7)(0.4) = 0.14 + 0.28 = +0.42

  4. Input Weight Gradients:

    ∂L∂W11=δ1[1]⋅x1=(−0.42)(2.0)=−0.84\frac{\partial L}{\partial W_{11}} = \delta_1^{[1]} \cdot x_1 = (-0.42)(2.0) = -0.84

    ∂L∂W12=δ1[1]⋅x2=(−0.42)(1.0)=−0.42\frac{\partial L}{\partial W_{12}} = \delta_1^{[1]} \cdot x_2 = (-0.42)(1.0) = -0.42

    ∂L∂b1=δ1[1]=−0.42\frac{\partial L}{\partial b_1} = \delta_1^{[1]} = -0.42

Every weight and bias gradient in an NN-layer neural network resolves through this identical scalar mechanic: downstream error ×\times incoming activation.