Backpropagation In-Depth – The Heart of Deep Learning

(Explained from absolute scratch to advanced level – 2025 understanding)

Why Backpropagation Exists

We want to train multi-layer neural networks (MLPs) using gradient descent, but the problem is:

How do we compute ∂L/∂w for a weight that is 5–10 layers deep inside the network?

Backpropagation (short for "backward propagation of errors") is the efficient algorithm that computes all these gradients using the chain rule in a smart way.
Invented by Rumelhart, Hinton, Williams in 1986 – still the foundation of all modern deep learning.

1. Forward Pass vs Backward Pass

2. Complete Mathematical Derivation (Step-by-Step)

Let’s take a simple 3-layer network:

Input → Hidden1 (ReLU) → Hidden2 (ReLU) → Output (Sigmoid)
Loss = Binary Cross Entropy

Notations:
- x = input
- y = true label (0 or 1)
- ŷ = predicted probability
- L = loss

Layer equations (forward):

z¹ = W¹x + b¹
a¹ = ReLU(z¹)

z² = W²a¹ + b²
a² = ReLU(z²)

z³ = W³a² + b³
ŷ = a³ = σ(z³) = 1/(1+e⁻ᶻ³)

Loss:
L = −[ y log ŷ + (1−y) log(1−ŷ) ]

Goal: Compute ∂L/∂W³, ∂L/∂b³, ∂L/∂W², … all the way back.

Step-by-Step Chain Rule (Backpropagation)

Step 1: Gradient w.r.t output (z³)

We need ∂L/∂z³

For BCE + Sigmoid, there is a beautiful simplification:

∂L/∂z³ = ŷ − y

Proof:
L = −y log σ − (1−y) log(1−σ)
∂L/∂ŷ = −y/ŷ + (1−y)/(1−ŷ)
And since ŷ = σ(z), dσ/dz = σ(1−σ)
So ∂L/∂z = (∂L/∂ŷ) × (dŷ/dz) = [−y/ŷ + (1−y)/(1−ŷ)] × ŷ(1−ŷ)
= (ŷ − y)

Magic! No need to compute separately.

Step 2: Gradient w.r.t W³ and b³

∂L/∂W³ = ∂L/∂z³ × (∂z³/∂W³) = (ŷ − y) ⋅ a²ᵀ
∂L/∂b³ = ŷ − y

Step 3: Back to hidden layer 2 (a²)

We need ∂L/∂a² to continue backward

∂L/∂a² = ∂L/∂z³ × ∂z³/∂a² = (ŷ − y) ⋅ W³ᵀ

Step 4: Back through ReLU in layer 2

∂L/∂z² = ∂L/∂a² ⊙ ReLU'(z²)
where ReLU'(z) = 1 if z>0 else 0

Step 5: Gradient w.r.t W² and b²

∂L/∂W² = ∂L/∂z² × a¹ᵀ
∂L/∂b² = ∂L/∂z²

Repeat same for layer 1.

General Backpropagation Rule (The 4 Equations You Must Memorize)

For any layer l:

1. δˡ = ∂L/∂zˡ (called "error term" or "delta")
2. ∂L/∂Wˡ = (aˡ⁻¹)ᵀ ⋅ δˡ
3. ∂L/∂bˡ = δˡ (sum over batch)
4. δˡ⁻¹ = (Wˡ)ᵀ ⋅ δˡ ⊙ g'(zˡ⁻¹) ← propagate error backward

This is repeated from output to input.

3. Backpropagation in Code – From Scratch (NumPy (Full Working Example)

import numpy as np

# Sigmoid and its derivative
def sigmoid(z): return 1 / (1 + np.exp(-z))
def sigmoid_prime(a): return a * (1 - a)

# ReLU
def relu(z): return np.maximum(0, z)
def relu_prime(z): return (z > 0).astype(float)

# Toy dataset: XOR
X = np.array([[0,0], [0,1], [1,0], [1,1]])
y = np.array([[0], [1], [1], [0]])

# Full Backpropagation Implementation from Scratch

class NeuralNetwork:
    def __init__(self):
        # Random init
        self.W1 = np.random.randn(2, 4) * 0.5
        self.b1 = np.zeros((1, 4))
        self.W2 = np.random.randn(4, 1) * 0.5
        self.b2 = np.zeros((1, 1))

    def forward(self, X):
        self.z1 = X @ self.W1 + self.b1
        self.a1 = relu(self.z1)
        self.z2 = self.a1 @ self.W2 + self.b2
        self.a2 = sigmoid(self.z2)  # output probability
        return self.a2

    def backward(self, X, y, lr=0.5):
        m = X.shape[0]

        # Forward pass (cache)
        self.forward(X)

        # === Output layer ===
        dz2 = self.a2 - y                    # (m,1)  ← magic BCE+sigmoid
        dW2 = self.a1.T @ dz2 / m             # (4,1)
        db2 = np.sum(dz2, axis=0, keepdims=True) / m

        # === Hidden layer ===
        da1 = dz2 @ self.W2.T                 # (m,4)
        dz1 = da1 * relu_prime(self.z1)       # (m,4)
        dW1 = X.T @ dz1 / m
        db1 = np.sum(dz1, axis=0, keepdims=True) / m

        # === Update weights ===
        self.W2 -= lr * dW2
        self.b2 -= lr * db2
        self.W1 -= lr * dW1
        self.b1 -= lr * db1

    def train(self, X, y, epochs=10000):
        for i in range(epochs):
            self.backward(X, y)
            if i % 1000 == 0:
                loss = -np.mean(y*np.log(self.a2 + 1e-8) + (1-y)*np.log(1-self.a2 + 1e-8))
                print(f"Epoch {i}, Loss: {loss:.4f}")

# Train XOR
nn = NeuralNetwork()
nn.train(X, y)

print("\nPredictions after training:")
print(nn.forward(X) > 0.5).astype(int)

Output after ~5000 epochs:

Predictions:
[[0]
 [1]
 [1]
 [0]]

Perfect XOR solved!

4. Modern Backpropagation (2025): What Changed?

But the core math is exactly the same!

5. Common Questions Answered

Q: Is backpropagation biologically plausible?
A: No. Brains don’t send errors backward. But it works amazingly well.

Q: Why is it called "backpropagation"?
A: Because we propagate the error δ backward through the layers.

Q: Vanishing/exploding gradients?
A: Happens when |λ| of Jacobian <<1 or >>1
Solutions: ReLU, LayerNorm, Residual connections, Gradient clipping

Q: Can we do it without chain rule?
A: No. Chain rule is mathematically inevitable.

Final Summary: The 4-Line Essence of Backpropagation

# This is literally all backprop is:
loss.backward()          # PyTorch: compute all gradients automatically
optimizer.step()         # update weights: W = W - lr * W.grad
optimizer.zero_grad()    # reset gradients to zero
model.train()               # forward pass again

But now you know exactly what happens inside loss.backward()!

You have now mastered backpropagation at both intuitive and mathematical levels. This knowledge will carry you through any neural network architecture invented in the next 20 years.

Happy training!