Reward is a specification

The optimiser answers the question you asked, which is rarely the question you meant.

U12.5Learning, and the capstoneUniversity35 min

Do this lesson in the simulator

An optimiser does what you asked. Not what you wanted, not what any reasonable person would have understood, and not what you would have written if you had thought about it for another ten minutes. What you asked.

So the reward function is not a hint or an encouragement. It is a specification of the behaviour, written in a language with no room for intent, and it will be read literally by something with more patience than you.

The failure has a name

Reward hacking, or specification gaming: the policy scores well and the behaviour is wrong. The published examples are worth knowing because they are all the same mistake.

  • A boat racing agent rewarded for points found a lagoon of respawning pickups, and drove in circles collecting them for ever instead of finishing the race.
  • A robot rewarded for the height of a block learned to flip the block over, putting its bottom face high, rather than lifting it.
  • A cleaning agent rewarded for not seeing mess learned to close its eyes.
  • A walker rewarded for forward velocity learned to fall forward repeatedly, which is fast and is not walking.

None of these are bugs in the optimiser. Every one is the optimiser doing its job on a specification that did not say what its author meant.

The two you will write yourself

Progress instead of arrival. Reward distance covered towards a goal and you get a robot that charges through the goal and keeps going, because stopping earns nothing. The fix is to score the end state, not the journey, or to score the journey and the end state together.

A proxy instead of the thing. Rewarding the sensor reading rather than the world behind it. A robot rewarded for a small depth reading can satisfy you by driving up against anything at all. A robot rewarded for seeing the tag can satisfy you by parking in front of it and never doing its job.

from bugbot import *
connect()

DT = 0.1

def home():
    for i in range(200):
        y = position()[1]
        if y < 3.0:
            break
        drive(-max(20, min(80, 1.2 * y)), 0, 0)
        wait(DT)
    stop()
    wait(0.5)

for cmd in (70, 100):
    y0 = position()[1]
    drive(cmd, 0, 0)
    wait(4.0)
    stop()
    wait(0.6)
    travel = position()[1] - y0
    print("command", cmd, ": progress", round(travel, 1), "cm, ended", round(abs(travel - 60), 1), "cm from the mark")
    home()

Run this in the simulator

The mark is 60 cm ahead. Command 100 wins on progress by a mile and misses the mark by a mile. If progress is your reward, that is the policy you will get, and the search that found it did nothing wrong.

Shaping, and how it goes wrong

A sparse reward, 1 for arriving and 0 otherwise, specifies the task perfectly and teaches almost nothing, because a random policy never arrives and every trial scores the same. So people add shaping: intermediate rewards that point the way.

Shaping is where the gaming gets in. Reward being near the goal and the policy learns to hover near it. Reward facing the goal and it learns to stare. There is a real theorem here worth knowing: shaping of the form F(s') - F(s), the difference of a potential function between states, leaves the optimal policy unchanged. Any other shaping can and usually does change what is optimal, which is another way of saying it changes the task you specified.

Writing one that survives

  • Score the end state. Where did it finish, was it stopped, was anything broken. These are hard to game because they are the thing you want.
  • Put the cost in the reward, not in your hopes. Time, energy, collisions, jerk. If you did not pay for it, the policy will spend it without limit.
  • Look at the winner, not at the number. A rising score is not evidence of anything until you have watched the behaviour that produced it. This is the single most useful habit in the whole field.
  • Expect to iterate. The first reward function is wrong. Watching how it gets gamed is how you find out what you actually meant, and that iteration is normal engineering, not a sign of failure.

Task: two scores, two winners

Try commands 55, 70, 85 and 100 for four seconds each from the same line, homing between trials. Print by progress:, the command that travelled furthest, and by error:, the command that finished nearest the mark 60 cm ahead. Then run the one the error score chose, and stop there.

from bugbot import *
connect()

DT = 0.1
MARK = 60.0
RUN_S = 4.0
COMMANDS = [55, 70, 85, 100]

Challenges

  1. Add a third score: nearest approach to the mark at any moment during the trial. Which command wins that, and why is it the worst of the three specifications?
  2. Write a score that wants the robot near the mark and stopped, and check that the command it picks is the same one.
  3. Describe, in two sentences, how a policy could score well on your combined reward while doing something you would refuse to accept.