Sim to real: why a policy that works in simulation fails on the robot

What a simulator leaves out, and what it does to a controller tuned inside one: friction, delay, noise and every robot being different. Tune a policy on a perfect robot, watch the same numbers fail with noise and delay turned on, then tune it across randomised robots. Every demo runs in the simulator.

Guidefree, runs in your browser

A policy is a rule for what to do next. Tune one in a simulator and it can look perfect: it arrives fast, it never overshoots, the chart goes straight to zero and stays there. Put the same numbers on a machine and it wanders, swings or sits there buzzing. The distance between the two worlds is called the reality gap, and closing it is most of the work in robot learning. On this page a small robot turns on the spot to face 90 degrees, and each demo is a program you can change and run.

The chart under each robot shows the error: how many degrees the robot still has to turn. A good policy takes that line to zero quickly and leaves it there. A policy that has met the reality gap does not.

What a simulator leaves out

A simulator is a model, and a model is a list of things somebody decided to include. The gap is everything that was left off that list.

• Friction and stiction. Real motors do nothing at all below some command, and the size of that dead patch changes with the mat, the temperature and the weight on the wheels.
• Delay. A command is worked out, sent over a radio link, read by a microcontroller and turned into current. The reading that started it was already old. Nothing in that chain is instant.
• Noise. Every measurement wobbles. Every measurement is also a little bit wrong in a fixed way, which is worse, because averaging does not remove it.
• Wear. A robot that has run for a term is not the robot that left the bench.
• Every robot is different. Two BugBots off the same reel have different motor gains, a different gyro bias and a different sideways leak. A policy fitted to one is a policy fitted to one.

This simulator is honest about some of that already. Each robot gets its own motor gains and its own gyro bias when the run starts, and set_noise() turns the wobble on its own sensors up and down. set_noise(0) gives a robot with perfect sensors, which no real robot has.

The policy

The policy on this page is two numbers. The robot reads the heading its own IMU believes, works out how far it still has to turn, and asks for a push:

push = KP × error + KD × (how fast the error is changing)

That is a PD controller, explained properly in the PID guide. Here it is just something with numbers to tune. One extra line matters for what follows: the motors ignore anything under about 15 percent, so while the robot is more than 3 degrees out the program sends at least 16.

To score a run, the program adds up how many degrees out the robot is over the last second and a half and takes the average. Lower is better. A score of 1 means it settled on the target. A score of 20 means it did not.

Tuned on a perfect robot

Perfect sensors, no delay, and motors exactly as strong as the model says. Raise KP until it arrives fast, add KD until the overshoot goes, and this is where you end up.

from bugbot import *
connect()

# the policy: the two numbers we are tuning
KP = 3.0
KD = 0.5

# the robot we are tuning on
set_noise(0)         # perfect sensors
DELAY = 0            # the motors feel a command the moment it is sent
SCALE = 1.0          # exactly as strong as the model says

DT = 0.1             # the loop runs 10 times a second
MIN = 16             # the smallest push that moves this robot

start = imu()[0]                   # the heading the robot believes now
target = (start + 90) % 360
pipe = [0.0] * DELAY
last = 90.0
score = 0.0
counted = 0
for tick in range(40):             # 4 seconds
    # degrees still to turn, -180 to 180
    error = (target - imu()[0] + 180) % 360 - 180
    change = (error - last) / DT
    last = error
    push = KP * error + KD * change
    if abs(error) > 3 and abs(push) < MIN:
        push = MIN if push > 0 else -MIN
    pipe.append(push)                     # the command goes in one end
    drive(0, 0, SCALE * pipe.pop(0))      # and comes out of the other
    plot("error", error)
    if tick >= 25:                        # score the last second and a half
        score = score + abs(error)
        counted = counted + 1
    wait(DT)
stop()
print("score:", round(score / counted, 2), "degrees out on average")

Nothing wrong with that. It is the right answer to the question that was asked. The question that was asked was about a robot that does not exist.

The same numbers on a real machine

We cannot put a real BugBot on this page, so the second run puts three of the missing things back by hand, and the simulator does the rest:

• set_noise(2.5) turns the wobble on the robot's own sensors up. The heading it reads is now a few degrees out, in a way that changes every reading.
• DELAY = 4 holds each command in a pipe for four ticks, so the motors feel it 0.4 seconds after the reading that caused it.
• SCALE = 1.4 makes this particular robot 40 percent stronger than the model said.

Both runs are in one program, so both lines land on the same chart.

from bugbot import *
connect()

KP = 3.0
KD = 0.5

DT = 0.1
MIN = 16

def trial(name, noise, delay, scale):
    set_noise(noise)
    start = imu()[0]
    target = (start + 90) % 360
    pipe = [0.0] * delay
    last = 90.0
    score = 0.0
    counted = 0
    for tick in range(40):
        error = (target - imu()[0] + 180) % 360 - 180
        change = (error - last) / DT
        last = error
        push = KP * error + KD * change
        if abs(error) > 3 and abs(push) < MIN:
            push = MIN if push > 0 else -MIN
        pipe.append(push)
        drive(0, 0, scale * pipe.pop(0))
        plot(name, error)
        if tick >= 25:
            score = score + abs(error)
            counted = counted + 1
        wait(DT)
    stop()
    wait(0.3)
    print(name, "score:", round(score / counted, 1))
    return score / counted

trial("perfect robot", 0, 0, 1.0)
trial("real machine", 2.5, 4, 1.4)

The same two numbers. One line settles, the other never does. Nothing in the program changed, nothing broke, and no error appeared. The policy is answering a question about a different robot.

It is worth knowing which part did the damage, because the three have different cures. Turn them on one at a time and the delay is nearly all of it: noise alone costs 3.2, stronger motors alone cost nothing, and the delay alone costs 24.5 out of the 28.3 that all three cost together.

By the time the motors feel a push, the robot has turned four ticks further than the reading that asked for it, so the push arrives pointing the wrong way, and a high gain makes that push large. Noise on its own only wobbles the robot, and stronger motors on their own barely show.

One thing to notice while comparing numbers: the same policy on the same machine scored 26 in the demo above and 28.3 in the runs behind that chart, because a trial never starts in exactly the state the last one ended in. A single score is a sample, not a result.

Measure the robot you have

The first thing to do about a gap is to stop guessing. Some of it can be measured directly. Ask the robot for a known push, watch how fast it actually turns, and compare that with what the model says. This is called system identification, and it turns a badly wrong model into a nearly right one.

The model says 100 percent of turn is 120 degrees per second, so 30 percent should be 36.

from bugbot import *
connect()

DT = 0.1
MIN = 16

# the real machine from the demo above
NOISE = 2.5
DELAY = 4
SCALE = 1.4

def trial(name, kp, kd):
    set_noise(NOISE)
    start = imu()[0]
    target = (start + 90) % 360
    pipe = [0.0] * DELAY
    last = 90.0
    score = 0.0
    counted = 0
    for tick in range(40):
        error = (target - imu()[0] + 180) % 360 - 180
        change = (error - last) / DT
        last = error
        push = kp * error + kd * change
        if abs(error) > 3 and abs(push) < MIN:
            push = MIN if push > 0 else -MIN
        pipe.append(push)
        drive(0, 0, SCALE * pipe.pop(0))
        plot(name, error)
        if tick >= 25:
            score = score + abs(error)
            counted = counted + 1
        wait(DT)
    stop()
    wait(0.3)
    print(name, "score:", round(score / counted, 1))

# hold a steady 30 percent and watch what the robot actually does
set_noise(NOISE)
pipe = [0.0] * DELAY
for tick in range(20):
    pipe.append(30)
    drive(0, 0, SCALE * pipe.pop(0))
    if tick == 5:                       # wait for the lag, then start the stopwatch
        h0 = imu()[0]
        t0 = clock()
    wait(DT)
turned = (imu()[0] - h0) % 360
rate = turned / (clock() - t0)
stop()
wait(0.5)
print("it turns", round(rate, 1), "deg/s at 30 percent, where the model says 36")
strength = rate / 36.0
print("so this robot is", round(strength, 2), "times as strong as the model")

trial("guessed", 3.0, 0.5)
trial("measured", 3.0 / strength, 0.5)

Measuring helped, and it did not fix it. That is the useful part. Identification can only find what it looks for, and the thing doing the damage here, the delay, is not visible in a steady turn rate. You can measure a gain in a minute. Measuring a delay needs a different experiment, and measuring the effect of a delay on a policy needs the policy.

Domain randomisation

If you do not know which world you will end up in, stop tuning for one. Pick the things you are unsure about, give each a range wider than you think reality needs, and score every candidate policy on a spread of worlds drawn from those ranges. Then keep the policy that does best across all of them.

This demo tries three gains on three robots each. One is the perfect robot from the first demo. One is slower than the model and slightly delayed. One is the noisy, delayed, strong machine from the second.

from bugbot import *
connect()

KD = 0.5
DT = 0.1
MIN = 16

def trial(name, kp, noise, delay, scale):
    set_noise(noise)
    start = imu()[0]
    target = (start + 90) % 360
    pipe = [0.0] * delay
    last = 90.0
    score = 0.0
    counted = 0
    for tick in range(40):
        error = (target - imu()[0] + 180) % 360 - 180
        change = (error - last) / DT
        last = error
        push = kp * error + KD * change
        if abs(error) > 3 and abs(push) < MIN:
            push = MIN if push > 0 else -MIN
        pipe.append(push)
        drive(0, 0, scale * pipe.pop(0))
        plot(name, error)
        if tick >= 25:
            score = score + abs(error)
            counted = counted + 1
        wait(DT)
    stop()
    wait(0.3)
    return score / counted

# noise, delay in ticks, how strong the motors are
ROBOTS = [(0, 0, 1.0), (1.0, 3, 0.8), (2.5, 4, 1.4)]

for KP in (3.0, 1.5, 1.0):
    name = "KP " + str(KP)
    scores = [trial(name, KP, *robot) for robot in ROBOTS]
    print(name, [round(s, 1) for s in scores],
          " average", round(sum(scores) / len(scores), 1),
          " worst", round(max(scores), 1))

The chart shows each gain three times over, once per robot, in the order they were run. KP 3 is the tuned answer from the first demo, and it is the worst of the three on average: 12.1, against 4.9 for KP 1.5 and 7.1 for KP 1.0. KP 1.5 gives up 2.4 points on the perfect robot and is the only gain that never scores worse than 9.6.

That trade is the whole idea. The randomised policy is worse than the tuned one in the tuned one's own world, and that is the price. What you buy with it is a policy whose behaviour you can predict on a robot you have not driven yet.

A robot it has never driven

Neither gain has seen this one: sensor noise at 1.5, 0.4 seconds of delay, and motors 30 percent weaker than the model, which is outside the range of strengths in the search.

from bugbot import *
connect()

KD = 0.5
DT = 0.1
MIN = 16

# a robot that was not in the search
NOISE = 1.5
DELAY = 4
SCALE = 0.7

def trial(name, kp):
    set_noise(NOISE)
    start = imu()[0]
    target = (start + 90) % 360
    pipe = [0.0] * DELAY
    last = 90.0
    score = 0.0
    counted = 0
    for tick in range(40):
        error = (target - imu()[0] + 180) % 360 - 180
        change = (error - last) / DT
        last = error
        push = kp * error + KD * change
        if abs(error) > 3 and abs(push) < MIN:
            push = MIN if push > 0 else -MIN
        pipe.append(push)
        drive(0, 0, SCALE * pipe.pop(0))
        plot(name, error)
        if tick >= 25:
            score = score + abs(error)
            counted = counted + 1
        wait(DT)
    stop()
    wait(0.3)
    print(name, "score:", round(score / counted, 1))

trial("tuned on the perfect robot", 3.0)
trial("tuned across three robots", 1.5)

What randomisation does not fix

Randomisation covers what you randomised over, and nothing past it. Change DELAY to 5 and SCALE to 1.5 in the demo above, a robot with half a second of lag and motors half as strong again as the model, and both gains fail together: 25.7 and 25.0. Widening the ranges costs something too. The wider you go, the more careful the winning policy becomes, and a policy chosen for a robot far worse than yours is slow on yours.

The other defences are worth more than they sound.

• Prefer feedback to a plan. A closed loop that measures as it goes survives a model that is 20 percent wrong. An open loop that drives for a worked-out number of seconds does not. If you only do one thing, do this one.
• Measure what you can. Identification is cheap and it moves the model most of the way, as the demo above shows.
• Keep measuring while it runs. A policy that re-estimates the robot as it drives closes the gap continuously rather than once.
• Hold out the conditions, not just the data. Tune at one noise level and report at another. One run is not a result: run it several times and report the spread as well as the average.

Lesson The reality gap is where that reporting side is taught properly, with a task to go with it.

Where this is taught

• Truth and belief: what the robot knows about itself, and why position() is not something a real robot has.
• Time and latency: where the delay in a loop comes from and how to measure it.
• Measuring your robot: the experiments that turn a guess into a number.
• A reading is a distribution: what noise on a sensor really means.
• Fitting a model: system identification, done properly.
• Generalisation: why a policy that fits its training conditions is not finished.
• Policy search: searching for the numbers instead of tuning them by hand.
• The reality gap: domain randomisation and honest evaluation, with a task.
• Tuning itself: the same search, at GCSE level.

Questions

What is the sim to real gap?

The difference between how a robot behaves in a simulator and how it behaves in the world. It comes from everything the simulator left out: friction, delay, sensor noise, wear, and the fact that every machine is slightly different from every other. A policy tuned in the simulator learned the task and the simulator together, and it cannot tell them apart.

Why does a policy trained in simulation fail on a real robot?

Because it was tuned against conditions that no longer hold. On this page a controller tuned on a perfect robot scores 0.5 there and 26 on a machine with sensor noise, 0.4 seconds of delay and stronger motors, where it overshoots by 44 degrees and never settles. Nothing crashed. It is answering the right question about the wrong robot.

What is domain randomisation?

Training or tuning across many simulated worlds instead of one, with the things you are unsure about drawn from ranges wider than you expect reality to need. The policy that wins is the one that does best on average across the range. It is a little worse in any single world than a policy tuned for that world alone, and much better on a world it has not seen.

Does domain randomisation always work?

No. It covers the range you randomised over, and nothing beyond it. Randomise delay up to 0.4 seconds and meet 0.8 seconds, and you are back where you started. Very wide ranges also cost performance, because the safest policy across a huge range is a timid one.

What is system identification in robotics?

Measuring the real machine to fix the model. Ask for a known command, measure what the robot actually does, and work out the number that connects them. On this page the robot turns 46.3 degrees per second where the model says 36, so it is 1.29 times as strong, and dividing the gain by that helps a little. It only fixes what it measures: the delay in the same demo is untouched by it.

Is feedback enough on its own?

It is the strongest single defence. A closed loop measures the error that the model got wrong and acts on it, so a model that is 20 percent out mostly makes the robot arrive late rather than wrong. It is not a licence to ignore the gap: the demos on this page are all closed loop, and the delay still breaks the keen gain, because feedback that acts on old information can push the wrong way.

How do you test a robot policy honestly?

Run it more than once, and report the spread as well as the average, because the first number you get is the one you stopped on. Report the worst case when the worst case is what matters. Tune under one set of conditions and report under another. Say what the simple hand-written baseline scores, because a learned policy that cannot beat ten minutes of hand tuning is not a result.

What is the difference between a simulator and a digital twin?

A simulator is a model of a kind of robot, built to be useful. A digital twin is a model of one particular machine, kept up to date from that machine's own measurements while it runs. A twin narrows the gap by identification, over and over, instead of once.

Learn it step by step

These lessons build the same ideas one at a time, each with tasks the simulator marks.

1. U1.3 Truth and belief The robot as a system, University
2. U1.4 Time, rate and latency The robot as a system, University
3. U1.6 Measuring your robot The robot as a system, University
4. U4.1 A reading is a distribution Noise and filtering, University
5. 8.6 Tuning itself Learning, Robot club
6. U12.2 Fitting a model to data Learning, and the capstone, University
7. U12.3 Generalisation Learning, and the capstone, University
8. U12.4 Policy search on a robot Learning, and the capstone, University
9. U12.6 The reality gap, and honest evaluation Learning, and the capstone, University