Types of machine learning explained

What supervised, unsupervised and reinforcement learning are each given, what each produces, and which robot jobs suit which. The same robot problem is solved all three ways, and every demo runs in the simulator.

Guidefree, runs in your browser

Machine learning comes in three kinds, and the difference between them is not the algorithm. It is what you are able to give the machine.

• Supervised learning is given examples with the right answers attached, and produces something that answers the same question about new examples.
• Unsupervised learning is given examples with no answers at all, and produces a description of how they fall into groups.
• Reinforcement learning is given no answers either, only a score after it acts, and produces a policy: what to do in each situation.

On this page the same small robot meets the same problem three times. It is on a 1.5 metre mat with four boxes on it, its job is to drive about without bumping into anything, and the only thing that changes is what it is given to learn from. Each demo below is a real program you can change and run.

Every demo uses the same two numbers as its view of the world. The robot's depth sensor returns 64 distances, and the program takes the two level rows of them, splits those sixteen readings down the middle, and keeps the nearest on each side:

def view():
    level = tof_grid()[16:32]                             # the two level rows
    left = min(min(level[0:4]), min(level[8:12]))         # nearest thing on the left
    right = min(min(level[4:8]), min(level[12:16]))       # and on the right
    return (min(120, left), min(120, right))

Two numbers in centimetres, both capped at 120 because the sensor answers 400 when it finds nothing at all. The lesson See as numbers takes the same grid apart reading by reading.

Supervised learning: examples with the answers on them

A person drove the robot to six places, looked at what was in front of it, and wrote down what it should do. That is the whole training set.

EXAMPLES = [
    ((35, 35), "go"),
    ((30, 60), "go"),
    ((60, 30), "go"),
    ((16, 45), "turn"),
    ((45, 16), "turn"),
    ((18, 18), "turn"),
]

To label a new view, find the example nearest to it and copy that example's word. This is nearest neighbour, the simplest supervised method there is, and the lessons Collecting data and Nearest neighbour build it up from a single reading.

from bugbot import *
import math
connect()

# six views a person drove the robot to and labelled by hand:
# how far it can see on its left and on its right, in cm, and what to do
EXAMPLES = [
    ((35, 35), "go"),
    ((30, 60), "go"),
    ((60, 30), "go"),
    ((16, 45), "turn"),
    ((45, 16), "turn"),
    ((18, 18), "turn"),
]
COLOUR = {"go": "green", "turn": "red"}

def view():
    # the two level rows of the depth grid, left half and right half
    level = tof_grid()[16:32]
    left = min(min(level[0:4]), min(level[8:12]))
    right = min(min(level[4:8]), min(level[12:16]))
    return (min(120, left), min(120, right))

def nearest(v):
    # the label of the example most like this view
    best, best_d = None, 1e18
    for feats, label in EXAMPLES:
        d = math.dist(feats, v)
        if d < best_d:
            best, best_d = label, d
    return best

trail = {"go": [], "turn": []}
bumps, was = 0, False
while clock() < 40:
    v = view()
    answer = nearest(v)
    for label in COLOUR:
        plot(label, min(math.dist(f, v) for f, l in EXAMPLES if l == label))
    x, y = position()
    trail[answer].append((75 + x, 75 + y))
    for label in COLOUR:
        draw(label, trail[label], COLOUR[label])
    if answer == "go":
        forward(60)
        wait(0.25)
    else:
        if v[0] > v[1]:
            turn_left(100)
        else:
            turn_right(100)
        wait(0.4)
    hit = bumped()
    if hit and not was:
        bumps += 1
    was = hit
stop()
print("bumps:", bumps)
print("go steps:", len(trail["go"]), " turn steps:", len(trail["turn"]))

The robot drives 312 cm in 40 seconds and never touches a box. On the mat, each dot is a step, green where it answered go and red where it answered turn.

Notice what has and has not been learned. The rule for when to turn was never written down by anybody: it comes out of six examples and one line of arithmetic, and you can change it by adding examples rather than by editing code. But the labels were a person's judgement. If that person had labelled (35, 35) as turn, the robot would be far more timid, and nothing in the data would say it was wrong. Supervised learning copies your answers, including your mistakes, which is why a classifier is always tested against examples it has never seen, as the lesson Generalisation sets out.

Supervised learning is most of the machine learning in the world: every spam filter, every photograph tagger, every model that reads handwriting. Its cost is always the same one. Somebody has to produce the answers.

Unsupervised learning: the same views, no labels

Now take the labels away. The robot drives about under a fixed rule, which nothing learned, and keeps 150 views. Then it splits them into two groups by k-means: start with two guesses for the centre of a group, put every reading with the nearer centre, move each centre to the average of the readings that chose it, and do it again.

from bugbot import *
import math
connect()

PASSES = 8          # how many times the two groups are worked out again

def view():
    level = tof_grid()[16:32]
    left = min(min(level[0:4]), min(level[8:12]))
    right = min(min(level[4:8]), min(level[12:16]))
    return (min(120, left), min(120, right))

# 1. drive about with a fixed rule and keep every reading. No labels.
readings, where = [], []
while clock() < 40:
    v = view()
    x, y = position()
    readings.append(v)
    where.append((75 + x, 75 + y))
    if min(v) < 25:
        if v[0] > v[1]:
            turn_left(100)
        else:
            turn_right(100)
        wait(0.4)
    else:
        forward(60)
        wait(0.25)
stop()
print("readings:", len(readings))

# 2. split them into two groups: k-means
centre = [readings[0], readings[-1]]
for p in range(PASSES):
    group = []
    for v in readings:
        a = math.dist(v, centre[0])
        b = math.dist(v, centre[1])
        group.append(0 if a < b else 1)
    for g in (0, 1):
        mine = [v for v, k in zip(readings, group) if k == g]
        if mine:
            centre[g] = (sum(v[0] for v in mine) / len(mine),
                         sum(v[1] for v in mine) / len(mine))
    plot("group A", min(centre[0]))
    plot("group B", min(centre[1]))
    wait(0.25)

for g in (0, 1):
    mine = [v for v, k in zip(readings, group) if k == g]
    close = sum(1 for v in mine if min(v) < 25)
    print("group", g, "centre", (round(centre[g][0]), round(centre[g][1])),
          "holds", len(mine), "readings,", close, "of them with something under 25 cm")
    draw("group " + "AB"[g], [p for p, k in zip(where, group) if k == g],
         "blue" if g == 0 else "orange")

The chart shows the two centres settling over the eight passes, one at about 59 by 74 cm and the other at 39 by 39. On the mat, blue dots are the places where the robot's view fell in the roomier group and orange dots where it fell in the tighter one, and the orange dots are the ones near the boxes and the edges.

So the grouping is real. It is also not the grouping you wanted. The boundary between the two sits at about 50 cm, not at the 25 cm where bumping becomes a risk, because k-means split the readings where the readings actually divide and nothing ever told it what the problem was. All it can say is that these 150 views came in two sorts. Which sort matters, and what to do about either, is still yours to decide: 16 of the 17 views with something under 25 cm landed in the tighter group, which is a fact you have to notice, not one the method announces.

That is unsupervised learning in general. It is at its best when you have a great deal of data and no answers, and the questions it is good at are: how many sorts of thing are in here, which readings are unlike all the others, and can I describe each reading with fewer numbers than I recorded. Grouping is called clustering, and the odd one out is anomaly detection, which is how a robot notices that today's motor current is not like any other day's.

Reinforcement learning: no answers, only a score

Now take away the examples as well. Nobody labels anything, nobody demonstrates anything, and the robot finds out what to do by trying things and being paid:

It keeps one number for every situation and action, Q[state][action], meaning how much reward it expects to collect from here on if it does that. There are two situations, near and far, and two actions, so it is four numbers, and they all start at zero. After every step, one of them moves a little towards what actually happened. That single line is Q-learning, and the lesson Learning by reward takes it apart term by term.

from bugbot import *
import random
connect()

# change these numbers and press Run
RATE = 0.3           # how far each surprise moves the table
DISCOUNT = 0.8       # how much the next situation's best value counts
EPSILON = 0.3        # the chance of a random action at the start
DECAY = 0.985        # epsilon is multiplied by this every step

random.seed(1)
ACTIONS = ["forward", "turn"]
STATES = ["near", "far"]
Q = {s: {a: 0.0 for a in ACTIONS} for s in STATES}

def state():
    # the nearest thing anywhere in front, in two buckets
    level = tof_grid()[16:32]
    return "near" if min(level) < 25 else "far"

def room_left():
    level = tof_grid()[16:32]
    return min(level[0:3] + level[8:11]) >= min(level[5:8] + level[13:16])

way = None
def act(a):
    global way
    if a == "forward":
        forward(60)
        wait(0.25)
        way = None
        return
    if way is None:                    # a new turn goes towards more room
        way = "left" if room_left() else "right"
    if way == "left":
        turn_left(100)
    else:
        turn_right(100)
    wait(0.2)

bumps, was = 0, False
epsilon = EPSILON
trail, hits = [], []
while clock() < 70:
    s = state()
    if random.random() < epsilon:
        a = random.choice(ACTIONS)     # explore
    else:
        a = max(Q[s], key=Q[s].get)    # use what it knows
    act(a)
    reward = 1.0 if a == "forward" else -0.2
    hit = bumped()
    x, y = position()
    if hit and not was:
        reward = -20.0
        bumps += 1
        hits.append((75 + x, 75 + y))
    elif hit:
        reward = -5.0
    was = hit
    s2 = state()
    # the learning rule: move this number towards what just happened
    Q[s][a] += RATE * (reward + DISCOUNT * max(Q[s2].values()) - Q[s][a])
    epsilon = epsilon * DECAY
    trail.append((75 + x, 75 + y))
    draw("path", trail, "blue", "line")
    draw("bumps", hits, "red", size=4)
    plot("bumps", bumps)
    plot("epsilon", epsilon)
stop()
print("bumps:", bumps)
for s in STATES:
    print(s, {a: round(v, 1) for a, v in Q[s].items()})

The table it prints at the end:

Close to something, turn. Otherwise, drive. Nobody said that, and nobody could have: there were no labelled examples of what to do and no demonstration to copy. The robot bumped into something once, 9.1 seconds in, that one number dropped, and it kept clear for the 61 seconds after it. It drives 668 cm in its 70 seconds, against the supervised robot's 312 cm in 40.

What it cost is the bump. A reinforcement learner has to do the wrong thing on purpose to find out that it is wrong, which is why so much of it is done in simulation. The second cost is the reward itself: it is a specification, and the robot will satisfy exactly what you wrote. Both are covered by the Q-learning guide and by the lesson Reward is a specification.

Which one for which job

Ask what you can honestly supply.

• You have answers, or you can get them. Use supervised learning. Naming what the camera is looking at, reading a sign, copying the way a person drove: all supervised. Most of the effort will go into collecting and labelling, not into the model.
• You have data and no answers. Use unsupervised learning, and expect to interpret the result yourself. Sorting a week of runs into the sorts of thing that happened, or spotting the reading that is unlike every other, needs no labels at all.
• Nobody knows the right answer, but you can tell good from bad afterwards. Use reinforcement learning. How to walk, how to balance, how to hold a speed on a surface nobody has measured. There is no table of right answers to copy, but there is a score.
• You are not sure the machine should be learning at all. Very often it should not. The kinematics of a robot come from geometry, a filter from a noise model, a controller from a gain you can tune in ten minutes. Learning earns its place when the thing you need is measurable but not derivable, or when it changes. The lesson Learning a behaviour is about that choice.

Real systems mix them. A robot that learns to drive from recorded human driving is supervised (it is copying labels), one that then improves by trial and reward is reinforcement, and the grouping of its sensor logs to find the situations worth recording more of is unsupervised. Two other names are worth knowing: semi-supervised learning, where a few examples are labelled and many are not, and self-supervised learning, where the answers are made out of the data itself, such as hiding part of a reading and training a model to fill it in. Both exist because labels are expensive and data is cheap.

Where this is taught

• See as numbers: the depth grid the three demos all read from.
• Collecting data: recording views and putting labels on them.
• Nearest neighbour: the supervised classifier used above, built up from one reading.
• A tiny network: weights trained from labelled examples, the other way to do supervised learning.
• Learning by reward: states, actions, rewards and the Q-learning rule.
• Project: situations: acting on what a classifier says, rather than only naming it.
• Learning a behaviour: the three settings side by side, and when learning is worth it at all.
• Policy search on a robot: learning from a score when there are no answers to copy.

Questions

What are the three types of machine learning?

Supervised learning, which is given examples with the right answers and learns to answer the same question about new ones. Unsupervised learning, which is given examples with no answers and finds the structure in them, usually groups. Reinforcement learning, which is given no answers at all, only a reward after each action, and learns what to do.

What is the difference between supervised and unsupervised learning?

Labels. Supervised learning is given the answer for every example and is judged on whether it gets new ones right. Unsupervised learning is given no answers, so there is nothing to be right about: it can only describe how the data falls apart. On this page the same views produced a robot that knows when to turn when they carried labels, and two unnamed groups when they did not.

Is reinforcement learning supervised or unsupervised?

Neither, and it is usually counted as a third kind. A supervised learner is told the right answer for each example. A reinforcement learner is never told the right action, only how well things went afterwards, and it has to work out for itself which of its actions earned the score.

What is an example of each type?

Supervised: a robot that names what its depth sensor is looking at from six labelled examples. Unsupervised: the same robot splitting 150 unlabelled views into two groups with k-means. Reinforcement: the same robot learning to turn near obstacles because bumping costs 20 and driving forward pays 1. All three are on this page and all three run.

Which type of machine learning should I use?

Whichever matches what you can supply. Answers, so use supervised. No answers but plenty of data, so use unsupervised and read the groups yourself. No answers but a score you can measure after the fact, so use reinforcement. And if you already have an equation for the thing, use the equation.

What is the difference between classification and clustering?

Classification puts a new thing into one of a set of named groups you defined, and it needs labelled examples. Clustering works out the groups itself from unlabelled data, and the groups come out without names or meanings. Nearest neighbour classifies. k-means clusters.

What is semi-supervised learning?

Learning from a small number of labelled examples together with a much larger number of unlabelled ones. It is common because labelling is the expensive part: the unlabelled data shows the model where the examples lie, and the few labels say what the regions mean.

What is deep learning, and where does it fit?

Deep learning is a family of models, not a fourth type. A deep neural network is a model with a great many parameters, trained by nudging them down their error, and it can be used in any of the three settings: supervised on labelled images, unsupervised or self-supervised on unlabelled text, or reinforcement to learn a policy from rewards. The lesson A tiny network is the smallest version of the same idea.

Do I always need labelled data for machine learning?

No, but you always need something to learn from. Supervised learning needs labels. Unsupervised learning needs only the data, and gives you less in return. Reinforcement learning needs no labels but does need a reward you can measure and a place where trying the wrong thing is allowed.

Are the types of machine learning on the GCSE or A level specification?

Pearson Edexcel GCSE Computer Science asks students to know about the issues raised by artificial intelligence, machine learning and robotics, and the three types are the usual way that is taught. AQA, OCR and Eduqas GCSE mention artificial intelligence and machine learning in the impacts of technology rather than as technical content. At A level the coverage varies by board, and the ideas here appear more often in project work than in exam questions.

Learn it step by step

These lessons build the same ideas one at a time, each with tasks the simulator marks.

1. 8.1 See as numbers Learning, Robot club
2. 8.2 Collecting data Learning, Robot club
3. 8.3 Nearest neighbour Learning, Robot club
4. 8.4 A tiny network Learning, Robot club
5. 8.5 Learning by reward Learning, Robot club
6. 8.8 Project: situations Learning, Robot club
7. U12.1 Learning a behaviour Learning, and the capstone, University
8. U12.4 Policy search on a robot Learning, and the capstone, University