The reality gap, and honest evaluation

Why a policy tuned in one world fails in another, and how to report what it does without lying.

U12.6Learning, and the capstoneUniversity40 min

Do this lesson in the simulator

A policy is only ever tuned in one world: a simulator, a quiet afternoon in the lab, one mat, one battery, one robot. It gets run in another. The distance between the two is the reality gap, and it is where most learned robot behaviour dies.

Why a tuned policy breaks

The policy did not learn the task. It learned the task and everything else that was true during training, and it cannot tell the two apart.

  • Unmodelled dynamics. Friction that changes with temperature, a battery that sags, backlash.
  • Sensor differences. Tuned against a clean reading, deployed against a noisy one. A threshold on a raw noisy signal fires early, because the first time the noise dips past the threshold counts as a crossing. A policy tuned at zero noise will make exactly this mistake and there was no way to see it coming from the training runs.
  • The robot is not that robot. Every BugBot has its own drive gains, its own gyro bias, its own sideways leak. A policy fitted to one is a policy fitted to one.
  • Different distribution of situations. Trained on an empty mat, deployed on a crowded one.

Domain randomisation

The standard answer is deliberately not to train in one world. Randomise what you are unsure of, over a range wider than you expect reality to occupy, and search for the policy that does best on average across all of them.

score = 0
for noise in (0.0, 1.0, 2.5):
    set_noise(noise)
    score += trial()

What comes out is more conservative than the policy tuned for any single world, and it is worse than that policy in its own world. That is the trade being made: some peak performance, in exchange for a policy whose performance you can predict.

The same reasoning as regularisation in U12.3, and the same as gain margin in control. You give up optimality against a model you do not fully believe, in exchange for robustness against the ways it is wrong.

What else closes the gap

  • Randomise, as above. Cheap and remarkably effective.
  • Identify, do not assume. Measure this robot's gains before the run. U12.2 in one paragraph, and it turns a badly wrong model into a nearly right one.
  • Adapt online. Keep estimating while running, and let the policy use the estimate. An adaptive controller closes the gap continuously rather than once.
  • Prefer feedback. A closed loop is tolerant of a model that is 20 percent wrong. An open loop is not. If you must choose one defence, choose this one.

Honest evaluation

The gap is also a reporting problem, and this part is entirely under your control.

One run is not a result. The first number you get is a sample from a distribution, and it is more likely to be a good one than a bad one, because you stop looking when it works. Run it n times and report the mean and the spread. With small n, report n as well, and report the failures rather than excluding them.

Report the worst case when the worst case is what matters. A parking policy that leaves a mean gap of 30 cm with a standard deviation of 12 is not a parking policy, however good the mean looks.

Hold out the conditions, not just the data. Tune at one noise level, report at another. Tune on one mat, report on a second. Anything else is measuring how well the policy fits the conditions it was tuned in, which you already knew.

Say what the baseline is. Learned beats hand-written, or it does not. A learning result without the obvious hand-written comparison is not a result, and the obvious comparison here is a feedback controller that took ten minutes to write.

from bugbot import *
connect()

DT = 0.1

def gap_now(n=8):
    vs = [distance() for i in range(n) if wait(DT) is None]
    return sum(vs) / len(vs)

print("gap now, quietly:", round(gap_now(), 1))
set_noise(3.0, depth=10.0)
readings = [gap_now(1) for i in range(12)]
mean = sum(readings) / len(readings)
spread = (sum((v - mean) ** 2 for v in readings) / len(readings)) ** 0.5
print("on a bad day:", round(mean, 1), "give or take", round(spread, 1))

Run this in the simulator

Same wall, same robot, and the readings are now scattered over a dozen centimetres. The mean is still roughly right, though it takes many more readings to pin it down, and that is the part people forget: the estimate of the mean got noisier too. The spread is the story, and a policy that brakes on one reading meets that spread on every run.

Task: tune it, then stress it

Tune the reading at which the robot starts braking, so that it ends up in the green band, 30 cm from the wall. Plot score as you tune. Then turn the noise up with set_noise(), run the tuned policy three times, and print mean: and spread: of the gaps it left. Put the noise back where it was for the run you hand in.

Start every trial from the same distance, and start it well clear of the threshold you are testing. A trial that begins two centimetres above the braking point triggers on the first unlucky reading and scores nonsense, and the search will believe it.

from bugbot import *
connect()

DT = 0.1
CRUISE = 85
GAP = 30.0
START_GAP = 55.0

Challenges

  1. Tune at set_noise(0) and evaluate at set_noise(3). How much worse is it than tuning with the noise on?
  2. Tune against the mean score across three noise levels at once. What does it cost you in the quiet world?
  3. Replace the learned threshold with a proportional controller on the gap error and compare both on mean and spread. Which would you ship?