Worksheet

U12.4 Policy search on a robot

Learning, and the capstone · University · about 40 min

What this lesson is about

Hill climbing on a real machine: a noisy score, a fair trial, and a budget measured in seconds.

• The setting
• Hill climbing
• The trial is where the rigour is
• Exploration and exploitation
• Why not just write the controller?

Questions 7 marks in all

1. [1 mark]This is the lesson's hill climb on a score whose true best value is 23. What does it print?

   def score(v):
       return abs(v - 23.0) + 2.0
   
   value, step = 0.0, 20.0
   best = score(value)
   for i in range(8):
       trial = value + step
       s = score(trial)
       if s < best:
           value, best = trial, s
       else:
           step = -step * 0.6
   print(round(value, 2), round(best, 2), round(step, 2))

2. [1 mark]Why is a derivative free search the right tool for tuning the lateral offset on the robot?

   1. AThe score comes out of a physical trial, not a formula, so there is no derivative to use
   2. BDerivative free methods always find the global optimum
   3. CDerivatives are too slow to compute for one parameter
   4. DThe offset has no effect on the score's slope

3. [1 mark]Each trial starts wherever the previous one ended. What is wrong with the scores?

   1. AEach score measures the previous policy as much as the current one
   2. BThey are noisier but still unbiased
   3. CNothing, as long as the trials are the same length
   4. DThe search will take fewer trials

4. [1 mark]A method needs 10,000 evaluations and each out and back trial takes 1.6 s of robot time. How many hours is that, to 2 decimal places?

5. [1 mark]The noise in one trial's score is about as large as the difference between two neighbouring offsets. What is the search doing?

   1. AFollowing coin flips; repeat and average the trials, or make them longer
   2. BConverging faster, because noise helps it explore
   3. CFinding the global optimum
   4. DNothing different, because the shrinking step handles noise

6. [1 mark]Which statements about exploration and exploitation are right?

   Tick every answer that is true.

   1. AHill climbing explores with large early steps and exploits as the step shrinks
   2. BAn upper confidence bound rule explores in proportion to how little an option has been tried
   3. CExploiting too early polishes a mediocre policy
   4. DA good enough method makes the trade-off disappear
   5. EEpsilon-greedy never takes a random action once it has a best option

7. [1 mark]Feeding flow()[0] back into the lateral command would cancel the leak without learning. What does the learned offset offer that feedback does not?

   1. AIt works with no sensor during the run and costs nothing at run time
   2. BIt adapts to changes in the leak that happen after tuning
   3. CIt removes the need for any trials
   4. DIt handles situations the learner never saw

The task: learn the offset that drives it straight

Hill climb the lateral offset using short out and back trials, plot score as you go, and then drive the robot at least 90 cm up into the green lane.

from bugbot import *
connect()

DT = 0.1
CMD = 60

Plan your program here, then type it in and press Run.

Do it on the robot
www.bugbotlab.com/learn/u12-4-policy-search/
The simulator checks it and tells you when it passes. Nothing to install, no account.

Challenges

1. Run the same search twice and compare the offsets it lands on. Is the difference smaller than the width of the lane?
2. Score each policy twice and average. How much of your budget did that cost, and did the search end up anywhere better?
3. Search over two numbers, an offset and a gain on flow()[0], by taking turns on each. What breaks first, the budget or the noise?