tldr
- A neural network layer without a nonlinearity is just an affine transformation. Composition of affine transformations is still an affine transformation. So adding new layers would be redundant, it would be equivalent to a single-layer network, a linear model.
- Linear models are probably not fit for images. Every pixel would be direct evidence of the image belonging to a certain class, with a linear relationship (the lighter the pixel the more likely it belongs to the class or vice versa).

Introduction

I’ve encountered this question multiple times early on in my career from people studying machine learning.

It seems that although it’s more or less intuitive how information propagates forward through a neural network, it’s less obvious what happens if you leave out nonlinearities.

So, this is my first attempt to contribute to the collective knowledge with my take on the explanation. I don’t expect this explanation to be better than all the others already available on the internet, but maybe this one additional example helps you connect the dots, and makes everything click.

It's worth noting that this article doesn’t discuss learning at all. It only concerns what functions a neural network can theoretically represent (e.g. if we set the weights manually). But if the network cannot represent a function, it cannot learn it either.

The best introductory course on machine learning in 2021, “Machine Learning” by Andrew Ng, of course, addresses this topic, but this part has always felt a bit shaky to me.

Lecture 8.1 - Neural Networks Representation | Non Linear Hypotheses - [Andrew Ng]

To recap the relevant part of the video:

The example was about a computer-vision car classifier (cars vs non-cars).
Prof. Ng picked 2 pixel locations on the image, and plotted the pixel intensity values for a few examples on a 2D graph, as a kind of slice of the high-dimensional data.
He pointed out that the plotted examples are not separable by a straight line.

from Machine Learning by Andrew Ng