Answers

U12.4 Policy search on a robot

Learning, and the capstone · University · about 40 min

What this lesson is about

Hill climbing on a real machine: a noisy score, a fair trial, and a budget measured in seconds.

• The setting
• Hill climbing
• The trial is where the rigour is
• Exploration and exploitation
• Why not just write the controller?

Questions 7 marks in all

1. [1 mark]This is the lesson's hill climb on a score whose true best value is 23. What does it print?

   def score(v):
       return abs(v - 23.0) + 2.0
   
   value, step = 0.0, 20.0
   best = score(value)
   for i in range(8):
       trial = value + step
       s = score(trial)
       if s < best:
           value, best = trial, s
       else:
           step = -step * 0.6
   print(round(value, 2), round(best, 2), round(step, 2))

   Answer:

   22.59 2.41 0.93

   It keeps stepping while the score falls, and each time it overshoots it turns round with a step 0.6 times smaller, closing in on 23 without ever using a derivative.

2. [1 mark]Why is a derivative free search the right tool for tuning the lateral offset on the robot?

   1. AThe score comes out of a physical trial, not a formula, so there is no derivative to use
   2. BDerivative free methods always find the global optimum
   3. CDerivatives are too slow to compute for one parameter
   4. DThe offset has no effect on the score's slope
   Answer: A. Hill climbing only compares scores. The price is that it finds a local optimum, which for one well behaved parameter is usually the global one.

3. [1 mark]Each trial starts wherever the previous one ended. What is wrong with the scores?

   1. AEach score measures the previous policy as much as the current one
   2. BThey are noisier but still unbiased
   3. CNothing, as long as the trials are the same length
   4. DThe search will take fewer trials
   Answer: A. Every trial must start from the same state. The out and back design returns the robot to where it began.

4. [1 mark]A method needs 10,000 evaluations and each out and back trial takes 1.6 s of robot time. How many hours is that, to 2 decimal places?

   Answer: 4.44 (accept within 0.01). 10,000 x 1.6 = 16,000 s, and 16,000 / 3600 = 4.44 hours, before any resets or battery changes. That is why simulation exists.

5. [1 mark]The noise in one trial's score is about as large as the difference between two neighbouring offsets. What is the search doing?

   1. AFollowing coin flips; repeat and average the trials, or make them longer
   2. BConverging faster, because noise helps it explore
   3. CFinding the global optimum
   4. DNothing different, because the shrinking step handles noise
   Answer: A. One trial cannot tell the two policies apart, so the comparisons are chance. Averaging costs budget, and you should know which you are spending.

6. [1 mark]Which statements about exploration and exploitation are right?

   Tick every answer that is true.

   1. AHill climbing explores with large early steps and exploits as the step shrinks
   2. BAn upper confidence bound rule explores in proportion to how little an option has been tried
   3. CExploiting too early polishes a mediocre policy
   4. DA good enough method makes the trade-off disappear
   5. EEpsilon-greedy never takes a random action once it has a best option
   Answer: A, B, C. The trade-off is a property of learning from your own actions and never goes away. Epsilon-greedy keeps a decaying fraction of random actions.

7. [1 mark]Feeding flow()[0] back into the lateral command would cancel the leak without learning. What does the learned offset offer that feedback does not?

   1. AIt works with no sensor during the run and costs nothing at run time
   2. BIt adapts to changes in the leak that happen after tuning
   3. CIt removes the need for any trials
   4. DIt handles situations the learner never saw
   Answer: A. Feedback handles changes the learner never saw. Real systems use both: learned feedforward for what is predictable, feedback for what is not.

The task: learn the offset that drives it straight

from bugbot import *
connect()

DT = 0.1
CMD = 60

from bugbot import *
connect()

DT = 0.1
CMD = 60
SETTLE = 3                         # ticks before the reading counts: the motors take a moment to come up

def turn_cmd():
    """Hold the robot square, so that sliding sideways is the only thing the trial measures."""
    err = (imu()[0] + 180) % 360 - 180
    if abs(err) < 2.0:
        return 0
    cmd = max(18.0, min(30.0, abs(1.5 * err)))     # below 15 the drive does nothing at all
    return -cmd if err > 0 else cmd

def leg(fwd, lat, seconds):
    """Drive for a moment and return how far the robot slid sideways, from the flow sensor."""
    slide = 0.0
    for i in range(SETTLE + int(seconds / DT)):
        drive(fwd, lat, turn_cmd())
        if i >= SETTLE:
            slide += flow()[0] * DT
        wait(DT)
    return slide

def score(offset):
    """One trial: out and back, so the robot ends where it started and the trial can be repeated."""
    out = leg(CMD, offset, 0.8)
    back = leg(-CMD, -offset, 0.8)
    stop()
    wait(0.2)
    return abs(out) + abs(back)

offset, step = 0.0, 16.0
best = score(offset)
for i in range(11):
    trial = offset + step
    s = score(trial)
    if s < best:
        offset, best = trial, s
    else:
        step = -step * 0.6         # wrong way, and getting close: turn round and take a smaller step
    plot("score", best)
    plot("offset", offset)
print("offset:", round(offset, 1))
print("score:", round(best, 2))

# the run itself, with the offset held on all the way
travelled = 0.0
for tick in range(300):
    drive(CMD, offset, turn_cmd())
    travelled += flow()[1] * DT
    if travelled > 105:
        break
    wait(DT)
stop()
wait(0.5)
print("travelled:", round(travelled, 1))