Gradient descent explained
What gradient descent is, what the loss, the gradient and the learning rate each do, and what a rate that is too small or too big looks like. A robot measures its own drive and fits a model of it in your browser: change the rate, press Run and watch the loss fall, crawl or explode.
Gradient descent is how a program finds the numbers that make a model fit. It measures how wrong the model is, works out which way each number should move to make it less wrong, and moves it a little that way. Then it does that again, a few hundred or a few million times. Every neural network you have heard of was trained like this, and so is the small model on this page. Here a robot measures its own drive, fits a model of it, and each demo below is a program you can change and run.
The first demo charts what the robot measured. The three after it chart the loss: one number for how wrong the model is. A good learning rate takes that line down to almost nothing in a few steps. A bad one crawls, or sends it up.
The idea in one line
new weight = old weight − learning rate × slope of the loss
- The weight is the number the program is trying to find. A network has millions of them. The model here has one.
- The loss says how wrong the model is on the data: one number, and smaller is better.
- The slope, or gradient, says how the loss changes when the weight changes. Negative slope means the loss falls as the weight grows, so the weight should grow.
- The learning rate says how much of that slope to take as a step.
The picture people use is walking downhill in fog: you cannot see the valley, but you can feel the ground slope under your feet, so you step downhill and feel again. The arithmetic is smaller than the picture, and the rest of this page works it out in full.
The model: how fast does this robot go
drive(60, 0, 0) asks for 60 percent. Nobody has told the robot what 60 percent means in centimetres per second, and it is not the same for every robot, or for the same robot on a different surface or a flatter battery. So measure it. The model has one weight, a:
speed in cm/s = a × command / 100
The robot measures its own speed with flow(), the optical sensor under it, which reports how fast the mat is sliding past. The drive takes about a quarter of a second to get up to speed, so each command is held for 0.8 s before anything is read. Each command is run forwards and then backwards, which leaves the robot where it started and cancels any fixed offset in the sensor.
The program
from bugbot import *
connect()
def speed_at(cmd):
for i in range(8): # 0.8 s: let the drive get up to speed
drive(cmd, 0, 0)
plot("command", cmd)
plot("speed", flow()[1])
wait(0.1)
total = 0.0
for i in range(3): # then three readings, 0.3 s
drive(cmd, 0, 0)
speed = flow()[1]
total = total + speed
plot("command", cmd)
plot("speed", speed)
wait(0.1)
return total / 3
data = []
for c in (30, 50, 70, 90):
v = (speed_at(c) - speed_at(-c)) / 2 # out and back, so it ends where it began
data.append((c / 100, round(v, 2)))
print("command", c, "->", round(v, 2), "cm/s, ", round(v / (c / 100), 2), "cm/s per unit")
stop()
print("data =", data)
Four pairs come back: 30 gives 5.15 cm/s, 50 gives 9.35, 70 gives 12.48 and 90 gives 16.77. The last column is the ratio, which is 17.17, 18.70, 17.83 and 18.63. If the four ratios were the same number, that number would be the answer and there would be nothing to fit. They are not, because the sensor is noisy and the drive is not perfectly linear, so the job is to find the one value of a that is least wrong about all four at once.
The loss
Take one measurement, x = 0.3 and v = 5.15. With a weight of a the model says a × 0.3, and the amount it is wrong by is a × 0.3 − 5.15. Square that, so that being too low and being too high both count as wrong, and add up the squares over all four measurements:
loss(a) = average of (a × x − v)² over the four measurements
At a = 0 the model says the robot never moves, and the loss is the average of 5.15², 9.35², 12.48² and 16.77², which is 137.73. That is the number the program starts from, and the whole job is to make it small.
The loss is a curve over the weight: one loss for every value a could take. Because the model is a straight line and the loss is a square, that curve is a parabola with one lowest point.
The gradient
The gradient is the slope of that curve at the weight you are standing on. For a squared loss it comes out as a short expression. Each measurement contributes
slope from one measurement = 2 × (a × x − v) × x
and the gradient is the average of those. Read it in three parts. (a × x − v) is the error: how wrong the model was, and which way. Multiplying by x says how much this weight was to blame: a weight that was multiplied by a small input had little to do with the output, so it is barely moved. The 2 comes from squaring.
At a = 0 the four errors are -5.15, -9.35, -12.48 and -16.77, and the gradient works out at -15.02. It is negative, so the loss falls as a grows, so a should grow. With a learning rate of 1 the first step is 0 − 1 × -15.02 = 15.02, and the loss there is 4.55, down from 137.73 in one step.
The program
from bugbot import *
connect()
LR = 1.0 # the learning rate: change this one number and press Run
STEPS = 30
def speed_at(cmd):
for i in range(8): # 0.8 s: let the drive get up to speed
drive(cmd, 0, 0)
wait(0.1)
total = 0.0
for i in range(3): # then three readings, 0.3 s
drive(cmd, 0, 0)
total = total + flow()[1]
wait(0.1)
return total / 3
data = []
for c in (30, 50, 70, 90):
v = (speed_at(c) - speed_at(-c)) / 2 # out and back, so it ends where it began
data.append((c / 100, v))
print("command", c, "->", round(v, 2), "cm/s")
stop()
a = 0.0 # the weight: cm/s for a command of 100
for step in range(STEPS + 1):
loss = sum((a * x - v) ** 2 for x, v in data) / len(data)
slope = sum(2 * (a * x - v) * x for x, v in data) / len(data)
plot("loss", loss)
plot("a", a)
if step % 5 == 0:
print("step", step, " a", round(a, 2), " loss", round(loss, 3), " slope", round(slope, 2))
a = a - LR * slope
wait(0.05)
print("speed =", round(a, 2), "x command / 100")
The weight goes 0, 15.02, 17.73, 18.22, 18.30, 18.32, and then stops moving, because at 18.32 the slope is zero: the bottom of the curve. The loss stops at 0.087 rather than 0, and that is not a failure of the method. It is the noise in the four readings. No straight line through the origin passes through all four points, and 0.087 is how far the best one misses by.
When the learning rate is too small
The rate is the one number you have to choose. Set it to 0.05 and every step is one twentieth of the slope.
The program
from bugbot import *
connect()
LR = 0.05 # the learning rate: change this one number and press Run
STEPS = 30
def speed_at(cmd):
for i in range(8): # 0.8 s: let the drive get up to speed
drive(cmd, 0, 0)
wait(0.1)
total = 0.0
for i in range(3): # then three readings, 0.3 s
drive(cmd, 0, 0)
total = total + flow()[1]
wait(0.1)
return total / 3
data = []
for c in (30, 50, 70, 90):
v = (speed_at(c) - speed_at(-c)) / 2 # out and back, so it ends where it began
data.append((c / 100, v))
print("command", c, "->", round(v, 2), "cm/s")
stop()
a = 0.0 # the weight: cm/s for a command of 100
for step in range(STEPS + 1):
loss = sum((a * x - v) ** 2 for x, v in data) / len(data)
slope = sum(2 * (a * x - v) * x for x, v in data) / len(data)
plot("loss", loss)
plot("a", a)
if step % 5 == 0:
print("step", step, " a", round(a, 2), " loss", round(loss, 3), " slope", round(slope, 2))
a = a - LR * slope
wait(0.05)
print("speed =", round(a, 2), "x command / 100")
Nothing goes wrong here. The loss falls every step and the answer is the same one, further away: it comes within 1 percent of 18.32 at step 111, where a rate of 1.0 got there in 5. On a model with one weight and four measurements that costs nothing. On a network with a million weights and a million examples it is the difference between an afternoon and a fortnight, which is why the rate is worth choosing well.
When the learning rate is too big
Now set it to 2.5. The slope still points the right way, but the step overshoots the bottom and lands further up the other side than it started. The next slope is bigger, so the next step is bigger, and it climbs out.
The program
from bugbot import *
connect()
LR = 2.5 # the learning rate: change this one number and press Run
STEPS = 30
def speed_at(cmd):
for i in range(8): # 0.8 s: let the drive get up to speed
drive(cmd, 0, 0)
wait(0.1)
total = 0.0
for i in range(3): # then three readings, 0.3 s
drive(cmd, 0, 0)
total = total + flow()[1]
wait(0.1)
return total / 3
data = []
for c in (30, 50, 70, 90):
v = (speed_at(c) - speed_at(-c)) / 2 # out and back, so it ends where it began
data.append((c / 100, v))
print("command", c, "->", round(v, 2), "cm/s")
stop()
a = 0.0 # the weight: cm/s for a command of 100
for step in range(STEPS + 1):
loss = sum((a * x - v) ** 2 for x, v in data) / len(data)
slope = sum(2 * (a * x - v) * x for x, v in data) / len(data)
plot("loss", loss)
plot("a", a)
if step % 5 == 0:
print("step", step, " a", round(a, 2), " loss", round(loss, 3), " slope", round(slope, 2))
a = a - LR * slope
wait(0.05)
print("speed =", round(a, 2), "x command / 100")
The weight goes 0, 37.56, -1.88, 39.53, -3.95, and on outwards, and the loss goes 137.7, 151.8, 167.4, 184.5, 203.5, up and up. This is divergence, and it is the usual reason a training run ends with a loss of nan: the numbers grow until they leave the range a computer can hold.
For this loss the tipping point can be worked out exactly. The curve's steepness is 2 × average of x², which for the four commands 0.3, 0.5, 0.7 and 0.9 is 0.82, and a step longer than 2 / 0.82 = 2.44 lands further from the bottom than it began. Try LR = 2.44, a whisker above it, and the weight sits near zero for all 30 steps while the loss creeps up to 144.5. Try LR = 2.4, a whisker below: the loss does fall, but the weight only reaches 11.4 in 30 steps, because each step nearly overshoots and most of it is thrown away. The fastest rates are well inside the limit, not next to it. For a real network nobody can work the limit out, so people try a few rates, watch the loss, and keep the largest one that falls smoothly.
When you do not need gradient descent
This particular model has an exact answer. The value of a that makes the loss smallest is the average of x × v divided by the average of x², which is 7.512 / 0.41 = 18.32: the same number the demo arrives at, in one line of arithmetic and no steps at all. The lesson Fitting a model to data does it that way, for a model with a slope and an offset, and that is the right tool whenever a problem allows it.
Gradient descent earns its place when there is no such formula. Put the weighted sum through tanh, stack a second layer on top, and the neat solution disappears, but the slope of the loss can still be worked out at whatever weights you are standing on. That is the whole reason a network can learn at all. The backpropagation guide is about how the slope is found for every weight in a network at once, and the neural network guide trains one on the robot's depth readings.
The words you will meet
- Batch gradient descent uses every example to work out one step, as the demos above do.
- Stochastic gradient descent uses one example at a time, and mini-batch uses a small handful, usually 32 or 64. Both take noisier steps, and both are far quicker when there are millions of examples, because a step does not need a pass over all of them.
- An epoch is one pass through all the training data.
- Momentum adds a fraction of the last step to this one, so the weights keep rolling in a direction that has been working.
- Adam is the optimiser most people reach for now. It keeps a separate step size for every weight, grown or shrunk from how that weight's gradient has been behaving.
- A local minimum is a dip that is not the lowest dip. The one-weight loss here cannot have one, since a parabola has a single bottom, but the loss of a network has many, and gradient descent stops in whichever one it walks into.
Where this is taught
- Fitting a model to data fits this same drive model with the exact least squares formula.
- Learning a behaviour sets out when a robot should learn something rather than have it programmed.
- A tiny network trains a network on the robot's depth readings by gradient descent.
- Tuning itself improves a number by trying it and keeping what worked, which is what you do when there is no gradient to follow.
- Policy search on a robot does the same for a whole behaviour, where the score comes from a trial rather than a formula.
Questions
What is gradient descent in simple terms?
It is a way of finding the numbers that make a model fit. Measure how wrong the model is, work out which way each number should move to make it less wrong, move each one a little that way, and repeat.
What is the gradient descent formula?
new weight = old weight − learning rate × gradient. For a squared loss the gradient of one weight is the average of 2 × error × input, where the error is what the model said minus what it should have said.
What is a loss function?
A rule that turns how wrong a model is into one number, so that training has something to make smaller. Squared error, used on this page, is the usual choice when the answer is a number. Cross entropy is the usual choice when the answer is a label.
What is the learning rate?
The size of the step, as a fraction of the gradient. Too small and training crawls; too big and the loss rises instead of falling. There is no formula for a good one on a real network, so people try a few and keep the biggest one whose loss still falls.
Why does my loss go up instead of down?
Almost always the learning rate is too big: each step overshoots the bottom and lands further up the far side. Divide it by ten and run again. If the loss goes up slowly rather than exploding, check the sign of the update, since adding the gradient instead of subtracting it climbs the hill on purpose.
Why does my loss become nan?
Divergence, usually. Once the weights are big enough the squares overflow, and the first infinity turns everything after it into nan. Lower the learning rate, and scale the inputs so they are around 1 rather than in the hundreds.
What is the difference between gradient descent and backpropagation?
Gradient descent is the rule for moving the weights once you know the gradients. Backpropagation is the method for working out those gradients in a network, one layer at a time, starting from the error at the output. They are used together, and neither is the other.
What is the difference between batch and stochastic gradient descent?
Batch uses every example to work out one step, so the step is accurate and slow. Stochastic uses one example at a time, so each step is cheap and noisy, and the noise turns out to help. Mini-batch, a few dozen at a time, is what nearly everything uses.
Does gradient descent always find the best answer?
For a loss with a single bottom, such as the parabola on this page, yes. For a network it finds a bottom, not the bottom, and which one depends on where the weights started. In practice the ones it finds are usually good enough, which is one of the surprises of the field.
How do you write gradient descent in Python?
Three lines inside a loop: work out the model's output, work out the gradient as the average of 2 × error × input, and subtract the learning rate times the gradient from the weight. The demos above are exactly that, with the data measured by a robot.
Learn it step by step
These lessons build the same ideas one at a time, each with tasks the simulator marks.
- U12.2 Fitting a model to data Learning, and the capstone, University
- U12.1 Learning a behaviour Learning, and the capstone, University
- 8.4 A tiny network Learning, Robot club
- 8.6 Tuning itself Learning, Robot club
- U12.4 Policy search on a robot Learning, and the capstone, University