Learning a behaviour

What learning buys over a hand-written rule, what it costs, and the smallest honest example.

U12.1Learning, and the capstoneUniversity30 min

Do this lesson in the simulator

Every module so far wrote the behaviour out by hand. The kinematics came from geometry, the filter from a noise model, the controller from a gain you chose and tuned. That is the right way to build a robot whenever you can, and most of a working robot is still built that way.

This module is about the cases where you cannot, and about what changes when the machine works the behaviour out from data instead.

What learning is, exactly

Three things, and if any of them is missing there is nothing to learn:

  1. A family of candidate answers. A gain between 0 and 5. A straight line with a slope and an intercept. A lookup table with sixty entries. This is the hypothesis class, and choosing it is a modelling decision you make, not something the data decides.
  2. A score that says how good a candidate is. A loss to be made small, or a reward to be made large. On a robot this is usually measured, which means it is noisy.
  3. A search. Arithmetic that solves for the best candidate in one go, or a loop that tries candidates and keeps the better ones.

Nothing else is required. A robot that drives a 4 second leg, measures how far it went, and divides is doing all three. That is the honest starting point, and the rest of the module is the same idea with more parameters and more care.

Three settings you will meet

Setting What the data is What comes out
Supervised learning inputs with the right answers attached a function from input to answer
Model learning (system identification) commands and what the robot then did a model you can predict with
Policy search, reinforcement learning trials and their scores a policy: what to do in a situation

Robotics uses all three, usually in the same system. U12.2 and U12.3 are the middle row, U12.4 and U12.5 the bottom one.

When learning is worth it

  • The thing you need is measurable but not derivable. Your robot's actual speed per unit of command, the sideways leak of this particular chassis, the friction of this particular mat. Nobody can calculate these for you.
  • The relationship is real but you have no equation for it. What the 64 depth readings mean about which way is passable.
  • It changes. A new mat, a flat battery, a worn motor. A model fitted this morning beats an equation from a data sheet printed last year.

When it is not

  • When you have the equation. Learning inverse kinematics from data, when U2 gives it to you exactly, is a waste of data and worse than the formula.
  • When failure is expensive. A learner explores, and exploring means doing the wrong thing on purpose. Choose carefully where that is allowed.
  • When you cannot measure the score. No score, no learning. This is the most common reason a learning project fails, and it usually fails silently: the score turns out to measure something other than what you wanted, which is U12.5.

About position() in this module

In U6 and U7 position() was forbidden, because a robot working out where it is may not peek at the answer. Here it is the lab's measuring rig: the overhead camera that supplies labels for training and scores for evaluation. That is exactly how robot learning is done in practice, with motion capture or a survey mark giving the ground truth that the robot itself never sees.

The rule that matters is the one that stays: the truth may inform the score, never the policy. A policy that reads position() is not a policy you can deploy. The capstone forbids it again for that reason.

The smallest example there is

from bugbot import *
connect()

# the data sheet: 20 cm/s at command 100
nominal = 20.0 * 60 / 100.0 * 4.0
x0, y0 = position()
forward(60)
wait(4.0)
stop()
wait(0.5)
x1, y1 = position()

measured = ((x1 - x0) ** 2 + (y1 - y0) ** 2) ** 0.5
print("the data sheet says", round(nominal, 1), "cm")
print("this robot went", round(measured, 1), "cm")
print("so its gain is", round(measured / nominal, 3))

Run this in the simulator

One parameter, one measurement, and the robot now predicts its own travel far better than the data sheet does. Every idea in this module is already present: a hypothesis class (a gain), a loss (the squared error between predicted and measured travel, whose minimiser happens to be that ratio), and a search (the division).

What is missing is any statement about how much to trust it. One run, one number, no idea of the spread. U12.3 and U12.6 are about fixing that.

Task: the data sheet is not your robot

Print nominal: the travel the data sheet predicts at command 60 for 4 seconds, measured: what this robot actually did, and gain: the ratio.

from bugbot import *
connect()

CMD = 60
SECONDS = 4.0
V_MAX = 20.0

Challenges

  1. Repeat the measurement five times and report the mean and the spread. How many runs before the mean stops moving?
  2. Measure the gain at command 30 as well as at 60. Is it the same number?
  3. Name one thing about this robot that a gain cannot capture, however carefully you measure it.