Worksheet

8.5 Learning by reward

Learning · Robot club · about 30 min

What this lesson is about

Q-learning: a table of how good each action is, filled in by trying.

• States and actions
• The table
• The reward, and the rule
• What it learns

Questions 8 marks in all

1. [1 mark]What makes learning by reward different from the classifiers earlier in the module?

   1. AThere are no labels: the robot tries actions and is told afterwards how well they went
   2. BIt needs more labelled samples
   3. CIt uses the camera instead of the depth grid
   4. DIt never makes mistakes while learning

2. [1 mark]The states are near (under 20 cm), mid (under 40 cm) and far, and the actions are forward and turn. How many numbers does the Q table hold?

3. [1 mark]What does this program print?

   def state(ahead):
       return "near" if ahead < 20 else ("mid" if ahead < 40 else "far")
   
   print(state(12))
   print(state(20))
   print(state(40))

4. [1 mark]What does this program print?

   Q = {"near": {"forward": 0.0, "left": 0.0}, "far": {"forward": 2.0, "left": 0.5}}
   s, a, s2 = "near", "forward", "far"
   reward = 1.0
   Q[s][a] += 0.3 * (reward + 0.8 * max(Q[s2].values()) - Q[s][a])
   print(round(Q[s][a], 2))

5. [1 mark]Why does the robot sometimes pick an action at chance instead of the best one it knows?

   1. AThose tries are how it discovers anything it does not already know
   2. BTo save battery
   3. CBecause the table is full
   4. DSo it bumps into things on purpose

6. [1 mark]What is was_bumped for in the reward loop?

   1. Abumped() stays true for a third of a second, so it makes one bump cost -20 only once
   2. BIt counts how many times the robot has turned
   3. CIt stops the program after the first bump
   4. DIt remembers which side the bump was on

7. [1 mark]After learning, what does the table usually say in the near state?

   1. ATurn is worth more than forward
   2. BForward is worth more than turn
   3. CBoth actions are worth zero
   4. DBoth actions are strongly negative

8. [1 mark]Why does a turn keep going the same way until the robot drives forward again?

   1. AA robot that picks left or right afresh every step can swap between them in a corner for ever
   2. BTurning left and right costs different amounts
   3. CThe depth grid can only see one side at a time
   4. DIt saves battery

The task: learn to wander

Let the robot learn for 88 seconds. It may bump early on. After 60 seconds it must not bump at all, and it must keep getting about the mat: rocking back and forth or spinning in one spot does not count. It must drive at least 250 cm in total. Print bumps: <n> at the end.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()
bumps = 0
# do this 880 times (tick counts from 0)
for tick in range(880):
    # drive forward at 60 (keeps going until the next command)
    forward(60)
    # pause 0.1 s (the robot keeps doing what it was told)
    wait(0.1)
    if bumped():
        bumps += 1
        # spin clockwise on the spot at 60
        turn_right(60)
        # pause 0.5 s (the robot keeps doing what it was told)
        wait(0.5)
# all motors off
stop()
print('bumps:', bumps)

Plan your program here, then type it in and press Run.

Do it on the robot
www.bugbotlab.com/learn/8-5-learning-by-reward/
The simulator checks it and tells you when it passes. Nothing to install, no account.

Challenges

1. Keep epsilon at 0.3 for the whole run. How many bumps now?
2. Give it three actions, forward, left and right, each choosing afresh. Watch what it does the first time it meets a corner.
3. Make a bump cost 0 instead of -20. What does the table say about forward in near now, and where does the robot end up?