Calculus I · When You Can’t Solve It, Chase It

When Newton breaks

A method you do not understand the failures of is a method you cannot trust. Newton has a clear weak spot, and it is written right into the formula: we divide by $f'(x_n)$. If the slope at our guess is close to zero — if the tangent line is nearly flat — then a nearly-horizontal line crosses the axis very far away, and the next guess hurls itself off into the distance instead of homing in. (Figure 9.2: a near-flat tangent throwing $x_{n+1}$ far from the root.)

The cleanest disaster: try to find the root of $f(x) = x^{1/3}$ (the cube root), whose only root is at $x = 0$. Its derivative is $f'(x) = \tfrac13 x^{-2/3}$. Run Newton’s formula and watch (this is a diamond problem below, but here is the punchline): every step gives

$$x_{n+1} = -2x_n.$$

The guess flips sign and doubles in size every single step: $1, -2, 4, -8, 16, \ldots$ It runs screaming away from the very root it was supposed to find. Same elegant method, same formula, complete failure — because the function’s slope misbehaves at the root.

The lesson is not “Newton is bad.” The lesson is that fast and reliable are different promises. Newton is the fastest method in this chapter when you start near a well-behaved root, and it is the most dangerous when you don’t. Hold that thought, because the slower method we meet next is slower precisely because it is built not to blow up.

Gradient descent: just walk downhill

Now the target changes. We do not want a place where $f$ is zero; we want the place where $f$ is smallest. (Remember the reframe from Chapter 7: a “loss” — a number measuring how wrong a model is — is just a function, and learning means making it as small as possible. Finding minima is the whole game.)

Here is the most intuitive idea in the book, and a 12-year-old and a professional ML engineer use the exact same picture. Imagine standing on the graph of $f$ in the dark. You cannot see the bottom of the valley. But you can feel which way the ground slopes under your feet — that is exactly what the derivative $f'(x)$ tells you. So take a step downhill: if the ground slopes up to the right ($f' > 0$), step left; if it slopes up to the left ($f' < 0$), step right. Either way, step against the slope. And take a bigger step where the slope is steeper. All of that is captured in one line:

$$\boxed{\;x_{n+1} = x_n - \eta\,f'(x_n)\;}$$

The minus sign is “go against the slope, i.e., downhill.” The number $\eta$ (the Greek letter eta) is how big a step you take per unit of slope — it is called the learning rate, and choosing it is the entire art. At the bottom of the valley the slope is zero, so the update stops moving: you have arrived. (Figure 9.3: a ball on a U-shaped curve taking downhill steps that shrink as it nears the bottom.)

The learning rate is everything — and we can prove it with algebra

Let us minimize the simplest possible valley, $f(x) = x^2$, whose bottom is obviously at $x = 0$. Its derivative is $f'(x) = 2x$, so the update rule becomes:

$$x_{n+1} = x_n - \eta\,(2x_n) = (1 - 2\eta)\,x_n.$$

Every step just multiplies the current position by the fixed number $(1 - 2\eta)$. But multiplying by the same number over and over is exactly a geometric sequence — the thing we studied all the way back in Chapter 1. So after $n$ steps, starting from $x_0$:

$$x_n = (1 - 2\eta)^n\,x_0.$$

And we already know, from Chapter 1, exactly when a geometric sequence shrinks to zero: when the thing being raised to the $n$th power has size less than $1$. So whether gradient descent works is decided entirely by the single number $|1 - 2\eta|$. Let us start at $x_0 = 1$ and turn the same crank for four different learning rates:

Read that table slowly, because it is the whole story of training a model in miniature. A small learning rate ($\eta = 0.1$) creeps down patiently and lands. A medium one ($\eta = 0.9$) overshoots the bottom each time but the overshoots shrink, so it spirals in. At the critical value ($\eta = 1$) it bounces between the same two points forever, getting nowhere. And a too-large rate ($\eta = 1.5$) overshoots so hard that each step lands farther out than the last, and the whole thing detonates. The condition for success is just $|1 - 2\eta| < 1$, which works out to $0 < \eta < 1$. (Figure 9.4: the four trajectories on the parabola — the smooth glide, the shrinking spiral, the stuck bounce, the explosion.)

There is one more magic value worth seeing: $\eta = \tfrac12$ makes the factor $1 - 2\eta = 0$, so $x_1 = 0$ — gradient descent reaches the exact minimum in a single step. Tuck that away. It is not a coincidence, and the next section explains it.

Toolkit: Gradient descent (finding a minimum)

To minimize $f$: 1. Pick a starting guess $x_0$ and a learning rate $\eta > 0$. 2. Repeat: $\quad x_{n+1} = x_n - \eta\,f'(x_n)$. 3. Stop when the steps become tiny (the slope is near zero — you’re at the bottom).

The learning rate is the dial that decides everything: too small and it crawls; too large and it overshoots and can explode. The sweet spot is the whole craft. Unlike Newton, gradient descent never divides by anything, so it can’t blow up from a flat spot — its failures come only from a step size you chose.

All 12 chapters of Calculus I · Foundations · All courses