Reward is a specification

The optimiser answers the question you asked, which is rarely the question you meant.

U12.5Learning, and the capstoneUniversity35 min

Do this lesson in the simulator

Print it: worksheet (PDF) · answers (PDF) · as a web page

Guide: Reward shaping explained

An optimiser does what you asked. Not what you wanted, not what any reasonable person would have understood, and not what you would have written if you had thought about it for another ten minutes. What you asked.

So the reward function is not a hint or an encouragement. It is a specification of the behaviour, written in a language with no room for intent, and it will be read literally by something with more patience than you.

The failure has a name

Reward hacking, or specification gaming: the policy scores well and the behaviour is wrong. The published examples are worth knowing because they are all the same mistake.

  • A boat racing agent rewarded for points found a lagoon of respawning pickups, and drove in circles collecting them for ever instead of finishing the race.
  • A robot rewarded for the height of a block learned to flip the block over, putting its bottom face high, rather than lifting it.
  • A cleaning robot rewarded for not seeing mess could learn to close its eyes. That one is a thought experiment from the safety literature rather than a result, and it is quoted so often because it is so plausible.
  • A walker rewarded for forward velocity learned to fall forward repeatedly, which is fast and is not walking.

None of these are bugs in the optimiser. Every one is the optimiser doing its job on a specification that did not say what its author meant.

Two overlapping sets: behaviour you meant and behaviour that scores well; the optimiser picks the best scorer, which may be outside what you meantwhat you meantwhat scores wellwhat youhoped forgood behaviourthe score missescircling for pickupsflipping the blockfalling forwardthe highest score:where it ends up
The optimiser only ever sees the right-hand set, and it goes to the highest point in it. When that point is outside what you meant, you get reward hacking, and no amount of extra search fixes it.

The two you will write yourself

Progress instead of arrival. Reward distance covered towards a goal and you get a robot that charges through the goal and keeps going, because stopping earns nothing. The fix is to score the end state, not the journey, or to score the journey and the end state together.

A proxy instead of the thing. Rewarding the sensor reading rather than the world behind it. A robot rewarded for a small depth reading can satisfy you by driving up against anything at all. A robot rewarded for seeing the tag can satisfy you by parking in front of it and never doing its job.

from bugbot import *
connect()

DT = 0.1

def home():
    for i in range(200):
        y = position()[1]
        if y < 3.0:
            break
        drive(-max(20, min(80, 1.2 * y)), 0, 0)
        wait(DT)
    stop()
    wait(0.5)

for cmd in (70, 100):
    y0 = position()[1]
    drive(cmd, 0, 0)
    wait(4.0)
    stop()
    wait(0.6)
    travel = position()[1] - y0
    print("command", cmd, ": progress", round(travel, 1), "cm, ended", round(abs(travel - 60), 1), "cm from the mark")
    home()

Run this in the simulator

The mark is 60 cm ahead. Command 100 wins on progress, 79.0 cm against 55.6, and misses the mark by 19.0 cm, where command 70 stops 4.4 cm short of it. If progress is your reward, that is the policy you will get, and the search that found it did nothing wrong.

Where the two trials finished against the mark 60 cm ahead0102030405060708090cm from the starting linethe mark, 60 cmcommand 70: progress 55.6 cm4.4 shortcommand 100: progress 79.0 cm19.0 past
Scored by progress, command 100 wins with 79.0 cm. Scored by how far it finished from the mark, command 70 wins by a long way. Same trials, two specifications, two different winners.

Shaping, and how it goes wrong

A sparse reward, 1 for arriving and 0 otherwise, specifies the task perfectly and teaches almost nothing, because a random policy never arrives and every trial scores the same. So people add shaping: intermediate rewards that point the way.

Shaping is where the gaming gets in. Reward being near the goal and the policy learns to hover near it. Reward facing the goal and it learns to stare. There is a real theorem here worth knowing: shaping of the form F(s') - F(s), the difference of a potential function between states, leaves the optimal policy unchanged. Any other shaping can and usually does change what is optimal, which is another way of saying it changes the task you specified.

Shaping rewards along two paths to the mark, and a reward for being near it that pays for hoveringF(s') - F(s), straight there+20+20+20total +60= F(end) - F(start)F(s') - F(s), with a loop on the way+20+20-20+20+20total +60= F(end) - F(start)reward for being near: +1 a tick within 5 cm of the mark+1+1+1 a tick, for everF = minus the distance to the mark, so a step closer earns what it gained
With a potential difference, the rewards along any path add up to F(end) minus F(start), so the loop's minus 20 cancels its plus 20 and circling earns nothing. A reward for being near has no such balance: hovering by the mark collects it every tick.

Writing one that survives

  • Score the end state. Where did it finish, was it stopped, was anything broken. These are hard to game because they are the thing you want.
  • Put the cost in the reward, not in your hopes. Time, energy, collisions, jerk. If you did not pay for it, the policy will spend it without limit.
  • Look at the winner, not at the number. A rising score is not evidence of anything until you have watched the behaviour that produced it. This is the single most useful habit in the whole field.
  • Expect to iterate. The first reward function is wrong. Watching how it gets gamed is how you find out what you actually meant, and that iteration is normal engineering, not a sign of failure.

Task: two scores, two winners

Try commands 55, 70, 85 and 100 for four seconds each from the same line, homing between trials. Print by progress:, the command that travelled furthest, and by error:, the command that finished nearest the mark 60 cm ahead. Then run the one the error score chose, and stop there.

from bugbot import *
connect()

DT = 0.1
MARK = 60.0
RUN_S = 4.0
COMMANDS = [55, 70, 85, 100]

Challenges

  1. Add a third score: nearest approach to the mark at any moment during the trial. Which command wins that, and why is it the worst of the three specifications?
  2. Write a score that wants the robot near the mark and stopped, and check that the command it picks is the same one.
  3. Describe, in two sentences, how a policy could score well on your combined reward while doing something you would refuse to accept.