Why does increasing the depth of a neural network often lead to performance degradation despite not being caused by overfitting?

Adding more layers can make it harder for the network to learn useful features because the gradient signal that trains the layers can become very weak or unstable as it travels back through many weights, a problem known as vanishing or exploding gradients; this makes the deeper layers harder to train even if the model is not overfitting. When the gradient is too small, the weights in the early layers change very little, so the network cannot improve its mapping and ends up performing worse than a shallower counterpart. Moreover, deeper architectures introduce more parameters that may not contribute to learning because the optimization landscape becomes more rugged, causing the optimizer to get stuck in suboptimal local minima. For example, a 10‑layer convolutional network on MNIST may achieve 99 % accuracy, while a 20‑layer version without special tricks can drop to 95 % simply because the deeper layers cannot be effectively trained. Residual connections or careful initialization are often needed to avoid this degradation.