Reward shaping explained
Why a sparse reward teaches a robot nothing, how shaping speeds learning up, and how a sensible looking reward gets gamed. Three searches with three rewards, all of them running in the simulator.
A learning robot is never told what to do. It is paid. You write a reward, it tries things, and it keeps whatever scored best. So the reward is not encouragement and it is not a hint. It is the specification of the behaviour, written in a language with no room for what you meant, and the robot will read it literally.
Two things go wrong. A reward that pays only for finishing the job teaches almost nothing, because a robot that has never finished the job has nothing to learn from. That is a sparse reward, and the usual fix is shaping: paying for progress along the way. But a shaped reward is a new specification, and a robot that finds a way to collect the payments without doing the job is doing exactly what you asked. That is reward hacking.
On this page a small robot has to stop on a mark 25 cm away, and a search tries fifteen policies to find out how. The only thing that changes between the demos is the reward. Each demo below is a real program you can change and run.
The chart under each demo has two lines. score is what the reward gave that trial, and ended is how far from the mark, in centimetres, the robot actually finished. Watch the two together. A reward worth having is one where the score going up means the robot getting better.
The job, the policy and the trial
The mark is 12 cm to the robot's right and 22 cm ahead, which is 25 cm away. The robot has omni wheels, so it can slide sideways without turning, and a policy here is three numbers:
forward how hard to drive forwards, -100 to 100
lateral how hard to drive sideways, -100 to 100
seconds how long to drive before stopping
A trial runs one policy for a 4 second window, whatever the policy does, and then hands back a score. A policy with seconds = 2.5 drives for 2.5 seconds and then sits still for 1.5. Between trials the robot drives itself back to the start, because a trial that begins wherever the last one ended measures the last policy as much as this one. Sixteen trials and their homing take about 100 seconds.
The score is worked out from position(), which is the simulator's overhead view of the mat and not something the robot carries. That is allowed, and it is how it is done in a lab: the truth may inform the score, but never the policy. A policy that reads position() is not a policy you could ever deploy. The lesson Learning a behaviour makes the same rule.
The search itself is hill climbing, one number at a time: change a parameter by a step, keep the change if the score improved, and if it did not, turn the step round and make it smaller. Fifteen trials, five for each of the three numbers. It is the search from Policy search on a robot, and none of what follows depends on it: any search would show the same thing, because the problem is in the reward.
A sparse reward teaches nothing
Here is the most obvious reward anyone would write. The job is to stop within 8 cm of the mark, so score 1 for doing that and 0 for everything else.
The program
from bugbot import *
import math
connect()
# change these and press Run
MARK = (12.0, 22.0) # the spot to stop on: 12 cm right and 22 cm ahead of the start
WINDOW = 4.0 # every trial lasts this long, whatever the policy does
TRIALS = 15
REWARD = "sparse" # "sparse", "shaped" or "progress"
DT = 0.1
def hold(fwd, lat):
# one tick of driving, with the heading held at 0
err = (heading() + 180) % 360 - 180
turn = 0.0 if abs(err) < 2.0 else (-20.0 if err > 0 else 20.0)
drive(fwd, lat, turn)
wait(DT)
def gap():
# how far the robot is from the mark, in cm
x, y = position()
return math.hypot(MARK[0] - x, MARK[1] - y)
def home():
# back to the start, so every trial begins in the same place
for i in range(80):
x, y = position()
if math.hypot(x, y) < 3.0:
break
a = math.atan2(-x, -y)
hold(100 * math.cos(a), 100 * math.sin(a))
stop()
wait(0.3)
def trial(policy):
fwd, lat, seconds = policy
closer, last = 0, gap()
for i in range(int(WINDOW / DT)):
if i * DT < seconds:
hold(fwd, lat)
else:
drive(0, 0, 0) # stopped, but the window keeps running
wait(DT)
d = gap()
if d < last - 0.05: # this tick took it closer to the mark
closer += 1
last = d
stop()
wait(0.3)
ended = gap()
home()
return ended, closer
def score_of(ended, closer):
if REWARD == "sparse":
return 1.0 if ended < 8.0 else 0.0
if REWARD == "progress":
return float(closer)
return -ended
LIMIT = [100.0, 100.0, WINDOW]
policy = [0.0, 0.0, 2.0] # what it starts from: sit still for 2 seconds
step = [40.0, 40.0, 1.0] # how big a change to try in each number
ended, closer = trial(policy)
best = score_of(ended, closer)
plot("score", best)
plot("ended", ended)
print("start", [round(p, 1) for p in policy], "ended", round(ended, 1), "closer", closer, "score", round(best, 1))
k = 0
for t in range(TRIALS):
trying = list(policy)
trying[k] = max(-LIMIT[k], min(LIMIT[k], policy[k] + step[k]))
ended, closer = trial(trying)
s = score_of(ended, closer)
if s > best:
policy, best = trying, s # better: keep it and carry on that way
else:
step[k] = -step[k] * 0.6 # worse: turn round, and take a smaller step
k = (k + 1) % 3 # take the three numbers in turn
plot("score", s)
plot("ended", ended)
print("trial", t, [round(p, 1) for p in trying], "ended", round(ended, 1), "closer", closer, "score", round(s, 1))
stop()
print("best policy", [round(p, 1) for p in policy], "score", round(best, 1))
print("clock", round(clock(), 1))
The score line is flat on zero for the whole run and the robot learns nothing at all. It is not that the search is weak. Look at trial 0: driving forwards at 40 finished 13.6 cm from the mark, much better than sitting still at 25.1 cm, and the reward said 0 for both. With nothing to tell the two apart, the search has no reason to keep the change, so it turns its step round, shrinks it, and spends the rest of the run creeping about near the policy it started from.
A sparse reward is a perfect specification and a useless teacher. It says exactly what you want and nothing about how to get there, so the only way to learn from it is to arrive by luck at least once. Work out the odds of that. One unit of forward command moves the robot 0.197 cm a second and one unit of lateral 0.141 cm a second, so with the drive time fixed at 3 seconds the policies that land within 8 cm of the mark fill about one fiftieth of the square of commands. Add the drive time as a third number to get wrong and it is far less than that.
A shaped reward gives it a slope
Shaping means paying for being closer, not only for arriving. The simplest version is to score the distance from the mark at the end of the window, and want it small. Nothing else changes: one word in the program.
The program
from bugbot import *
import math
connect()
# change these and press Run
MARK = (12.0, 22.0) # the spot to stop on: 12 cm right and 22 cm ahead of the start
WINDOW = 4.0 # every trial lasts this long, whatever the policy does
TRIALS = 15
REWARD = "shaped" # "sparse", "shaped" or "progress"
DT = 0.1
def hold(fwd, lat):
# one tick of driving, with the heading held at 0
err = (heading() + 180) % 360 - 180
turn = 0.0 if abs(err) < 2.0 else (-20.0 if err > 0 else 20.0)
drive(fwd, lat, turn)
wait(DT)
def gap():
# how far the robot is from the mark, in cm
x, y = position()
return math.hypot(MARK[0] - x, MARK[1] - y)
def home():
# back to the start, so every trial begins in the same place
for i in range(80):
x, y = position()
if math.hypot(x, y) < 3.0:
break
a = math.atan2(-x, -y)
hold(100 * math.cos(a), 100 * math.sin(a))
stop()
wait(0.3)
def trial(policy):
fwd, lat, seconds = policy
closer, last = 0, gap()
for i in range(int(WINDOW / DT)):
if i * DT < seconds:
hold(fwd, lat)
else:
drive(0, 0, 0) # stopped, but the window keeps running
wait(DT)
d = gap()
if d < last - 0.05: # this tick took it closer to the mark
closer += 1
last = d
stop()
wait(0.3)
ended = gap()
home()
return ended, closer
def score_of(ended, closer):
if REWARD == "sparse":
return 1.0 if ended < 8.0 else 0.0
if REWARD == "progress":
return float(closer)
return -ended
LIMIT = [100.0, 100.0, WINDOW]
policy = [0.0, 0.0, 2.0] # what it starts from: sit still for 2 seconds
step = [40.0, 40.0, 1.0] # how big a change to try in each number
ended, closer = trial(policy)
best = score_of(ended, closer)
plot("score", best)
plot("ended", ended)
print("start", [round(p, 1) for p in policy], "ended", round(ended, 1), "closer", closer, "score", round(best, 1))
k = 0
for t in range(TRIALS):
trying = list(policy)
trying[k] = max(-LIMIT[k], min(LIMIT[k], policy[k] + step[k]))
ended, closer = trial(trying)
s = score_of(ended, closer)
if s > best:
policy, best = trying, s # better: keep it and carry on that way
else:
step[k] = -step[k] * 0.6 # worse: turn round, and take a smaller step
k = (k + 1) % 3 # take the three numbers in turn
plot("score", s)
plot("ended", ended)
print("trial", t, [round(p, 1) for p in trying], "ended", round(ended, 1), "closer", closer, "score", round(s, 1))
stop()
print("best policy", [round(p, 1) for p in policy], "score", round(best, 1))
print("clock", round(clock(), 1))
Now every trial says something. The first change, driving forwards at 40, is worth 11.5 cm and is kept. Sliding right at 40 is worth another 5.5. Driving for 3 seconds instead of 2 is worth another 5.2. By trial 13 the robot stops 0.3 cm from the mark, with the policy forward 40, lateral 31.4, for 3.0 seconds.
That is what shaping buys: a score that changes when the robot changes, everywhere in the space, not only at the finish. The same idea appears in every kind of learning from rewards. In Q-learning it is the difference between paying a robot only when it completes a lap and paying it a little for every step forward.
Shaping is not free. You have written a new specification, and the robot will now optimise that one instead of the job. The safest shaped rewards are the ones that still describe the end state, like the distance left here. The dangerous ones are the ones that pay for activity.
A reward that looks fine and is not
Here is a reward that many people write. Paying for the final distance means most of a trial earns nothing, so pay the robot for making progress: count the ticks in which it got closer to the mark. It is positive, it says something about almost every tick, and it sounds like exactly what you want.
The program
from bugbot import *
import math
connect()
# change these and press Run
MARK = (12.0, 22.0) # the spot to stop on: 12 cm right and 22 cm ahead of the start
WINDOW = 4.0 # every trial lasts this long, whatever the policy does
TRIALS = 15
REWARD = "progress" # "sparse", "shaped" or "progress"
DT = 0.1
def hold(fwd, lat):
# one tick of driving, with the heading held at 0
err = (heading() + 180) % 360 - 180
turn = 0.0 if abs(err) < 2.0 else (-20.0 if err > 0 else 20.0)
drive(fwd, lat, turn)
wait(DT)
def gap():
# how far the robot is from the mark, in cm
x, y = position()
return math.hypot(MARK[0] - x, MARK[1] - y)
def home():
# back to the start, so every trial begins in the same place
for i in range(80):
x, y = position()
if math.hypot(x, y) < 3.0:
break
a = math.atan2(-x, -y)
hold(100 * math.cos(a), 100 * math.sin(a))
stop()
wait(0.3)
def trial(policy):
fwd, lat, seconds = policy
closer, last = 0, gap()
for i in range(int(WINDOW / DT)):
if i * DT < seconds:
hold(fwd, lat)
else:
drive(0, 0, 0) # stopped, but the window keeps running
wait(DT)
d = gap()
if d < last - 0.05: # this tick took it closer to the mark
closer += 1
last = d
stop()
wait(0.3)
ended = gap()
home()
return ended, closer
def score_of(ended, closer):
if REWARD == "sparse":
return 1.0 if ended < 8.0 else 0.0
if REWARD == "progress":
return float(closer)
return -ended
LIMIT = [100.0, 100.0, WINDOW]
policy = [0.0, 0.0, 2.0] # what it starts from: sit still for 2 seconds
step = [40.0, 40.0, 1.0] # how big a change to try in each number
ended, closer = trial(policy)
best = score_of(ended, closer)
plot("score", best)
plot("ended", ended)
print("start", [round(p, 1) for p in policy], "ended", round(ended, 1), "closer", closer, "score", round(best, 1))
k = 0
for t in range(TRIALS):
trying = list(policy)
trying[k] = max(-LIMIT[k], min(LIMIT[k], policy[k] + step[k]))
ended, closer = trial(trying)
s = score_of(ended, closer)
if s > best:
policy, best = trying, s # better: keep it and carry on that way
else:
step[k] = -step[k] * 0.6 # worse: turn round, and take a smaller step
k = (k + 1) % 3 # take the three numbers in turn
plot("score", s)
plot("ended", ended)
print("trial", t, [round(p, 1) for p in trying], "ended", round(ended, 1), "closer", closer, "score", round(s, 1))
stop()
print("best policy", [round(p, 1) for p in policy], "score", round(best, 1))
print("clock", round(clock(), 1))
The two lines on the chart go opposite ways, and that is the whole lesson. The score climbs from 24 on the first trial to 38 on the last, while the robot finishes four times further from the mark than it did at trial 2.
Read the printed trials. At trial 2 the policy forward 40, lateral 40, for 3.0 seconds stopped 2.9 cm from the mark and scored 30. At trial 6 the search cut the forward command from 40 to 16, which finished 15.4 cm away, and the score went up to 33, so it kept the worse robot. By trial 14 the policy is forward 16, lateral 31.4, for 3.7 seconds: a crawl that drives for nearly the whole window, finishes 11.9 cm short, and scores 38.
Nothing has gone wrong with the search. It found the best policy for the reward it was given. Counting ticks that got closer pays for time spent approaching, and the fastest way to spend a lot of time approaching is to never arrive. Worse, arriving is punished: once the robot is on the mark and stopped, every remaining tick earns nothing at all, so the policy that does the job exactly is beaten by one that dawdles.
Why sensible rewards get hacked
Every one of these has the same shape. The reward measures something that goes with the job in the cases you imagined, and the optimiser finds the case you did not.
- Paying for progress instead of arrival. As above. A robot paid for distance covered towards a goal drives through it and keeps going, because stopping earns nothing.
- Paying for a sensor instead of the world. A robot paid for a small depth reading can satisfy you by driving up against anything at all. A robot paid for seeing a tag can park in front of it and never do its job.
- Paying for an action instead of a result. In the Q-learning wanderer, paying for the command
forwardrather than for actually moving gives a robot that presses into a corner for the rest of the run, scoring well and going nowhere. - Paying twice for the same ground. A reward for getting closer, with no charge for getting further away, pays a robot to drive towards the goal and back out again for ever. It is the same trap as counting ticks, and it is why rewards for progress are usually written as the change in distance, with the negatives counted too, so that a round trip is worth exactly zero.
The published examples in research are the same mistake at a larger scale. A boat racing agent paid for points found a lagoon of pickups that came back, and drove in circles collecting them instead of finishing the race. A robot arm paid for the height of a block's bottom face learned to flip the block over rather than lift it. A walker paid for forward velocity learned to fall over forwards, repeatedly, which is fast and is not walking.
Writing a reward that survives
- Score the end state. Where did it finish, was it stopped, did it break anything. These are hard to game because they are the thing you want.
- Shape towards the end state, not away from it. The distance left at the end is a shaped score that still describes the job. A count of nice moments during the trial is not.
- Make a round trip worth nothing. If you pay for progress, pay the change in distance and charge for the reverse, so the robot cannot earn by going back and forth.
- Charge for what you do not want. Time, energy, bumps, jerk. Anything you do not pay for, the policy will spend without limit.
- Watch the robot, not the score. A rising score is not evidence of anything until you have seen the behaviour that produced it. In the third demo the score rose all the way to a robot that does not do the job.
- Expect to rewrite it. The first reward is wrong. Watching how it gets gamed is how you find out what you actually meant.
Where this is taught
- Learning by reward: states, actions, rewards and the Q-learning rule on this robot.
- Tuning itself: a robot trying settings and keeping the best, which is a search with a score.
- Learning a behaviour: what learning needs, and why the truth may inform the score but never the policy.
- Policy search on a robot: trials, hill climbing, and why a trial you cannot trust is worse than no trial.
- Reward is a specification: reward hacking, progress against arrival, and rewards that survive contact with an optimiser.
- Capstone: the whole robot: a job scored end to end, where a hacked reward shows up as a robot that passes and is useless.
Questions
What is reward shaping?
Adding intermediate rewards to a task that would otherwise pay only at the finish, so that a learner gets useful feedback from trials that do not finish. On this page the sparse reward paid 1 for stopping within 8 cm of a mark and taught nothing in sixteen trials, while a shaped reward of minus the distance left took the robot from 25 cm away to 0.3 cm in thirteen.
Why are sparse rewards hard to learn from?
Because almost every trial scores the same, so nothing tells the learner which change was an improvement. It has to stumble on success by chance before it can start climbing. With the drive time fixed, fewer than one policy in fifty here lands close enough to score at all, and a learner that never scores never learns.
What is reward hacking?
When a policy scores well and the behaviour is wrong, because the reward did not say what its author meant. It is the optimiser doing its job on a faulty specification, not a bug. On this page, paying a point for each tick in which the robot got closer produced a crawl that stops 11.9 cm short and scores better than the policy that lands on the mark.
How do you stop a robot gaming its reward?
Score the end state you actually want, charge for the costs you do not want spent, make any progress term cancel out over a round trip, and watch the behaviour rather than the number. Then assume you have missed something and look at the winner before you believe the score.
What is the difference between a sparse and a dense reward?
A sparse reward is almost always zero, and says something only when the task succeeds or fails. A dense, or shaped, reward says something after every step or every trial. Sparse rewards specify the job cleanly and are hard to learn from. Dense rewards are easy to learn from and are easy to get wrong.
Does shaping change what the robot learns?
It can. Adding payments changes which policy scores best, so a careless shaping term can make some other behaviour the winner. The safe kind is a term that cancels out over any round trip, usually written as the change in a distance, since then the best behaviour is the same as without it. A term that pays for activity, like counting good moments, does not cancel and can be farmed.
What is the difference between a reward and a loss?
They are the same thing with opposite signs. A reward is something to make as large as possible and a loss something to make as small as possible, so minus a loss is a reward. The shaped score on this page is minus the distance left, which is a reward built from a loss.
Is reinforcement learning on the GCSE or A level specification?
Not by name. None of the GCSE or A level Computer Science specifications (AQA, OCR, Pearson Edexcel, Eduqas) ask for reinforcement learning or reward functions. Pearson Edexcel GCSE does ask about the issues raised by artificial intelligence and machine learning, and a robot that scores well while doing the wrong thing is a concrete example to discuss. It also makes a good A level programming project: a policy, a trial, a score and a search fit in about sixty lines.
Learn it step by step
These lessons build the same ideas one at a time, each with tasks the simulator marks.
- 8.5 Learning by reward Learning, Robot club
- 8.6 Tuning itself Learning, Robot club
- U12.1 Learning a behaviour Learning, and the capstone, University
- U12.4 Policy search on a robot Learning, and the capstone, University
- U12.5 Reward is a specification Learning, and the capstone, University
- U12.7 Capstone: the whole robot Learning, and the capstone, University