AI, robots and bias

What AI is good for, where it goes wrong, and auditing a training set for bias.

F12.7Technology and societyGCSE20 min

Machines now make decisions that used to be made by people: who gets an interview, what a camera thinks it sees, when a car brakes. Those decisions are learned from data, and data carries the marks of the world it came from. This lesson is about what AI and robots are good for, what goes wrong, and how to check a training set before trusting what it teaches.

What AI and machine learning are

Artificial intelligence is software that does things that normally need human intelligence: recognising a face, understanding speech, planning a route. Machine learning is the usual way to build it: instead of writing the rules, you show the program many examples and it finds the patterns itself. That is what module 8 did with the robot.

Because the rules are learned rather than written, nobody can simply read them, which is why these systems are hard to check.

Where they help

doing dangerous work: inspecting reactors, clearing mines, deep sea repair;
doing dull and repetitive work faster and more consistently than a person;
spotting patterns people miss, such as early signs of disease in a scan;
helping people: speech recognition, live captions, prosthetics, self-driving for those who cannot drive.

Where they go wrong

Bias. A model learns from its training data. If the data mostly shows one kind of person, the model works worse for everyone else: face recognition that fails on darker skin, voice recognition that struggles with regional accents, a hiring model that learned from a company's past hires and repeats its past unfairness.

No explanation. If a model refuses your loan, there may be no answer to "why?".

Responsibility. When a self-driving car hurts someone, who is accountable: the owner, the manufacturer, the programmer, the person who chose the training data? The law is still catching up.

Jobs. Automation removes some jobs, changes others, and creates new ones that need different skills. The gain and the loss rarely fall on the same people.

Data hunger. These systems need huge amounts of data, often personal, which brings us back to lesson F12.2.

Auditing a training set

The first check on any model is whether its training data looks like the world it will be used in. That is just counting:

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

samples = ["day", "day", "day", "day", "day", "day", "night", "night"]
counts = {}
for s in samples:
    counts[s] = counts.get(s, 0) + 1
for name, n in counts.items():
    share = n / len(samples) * 100
    print(f"{name}: {n} ({share:.0f}%)")
print("The robot will see night far less often in training than on the mat.")

Run this in the simulator

A model trained mostly on bright daylight will do badly at dusk, and it will not warn you: it will just be confidently wrong.

Task: audit the training set

Count how many samples of each label are in samples. Print <label>: <n> (<share>%) for each, sorted from the most common to the least, with the share rounded to the nearest whole number. Any label making up less than 15% is under-represented: add UNDER-REPRESENTED to its line. At the end print labels: <n> and under-represented: <labels>, the under-represented labels joined by ,.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

samples = [
    "day", "day", "day", "day", "day", "day", "day", "day", "day", "day",
    "day", "day", "day", "dusk", "dusk", "night", "night", "night", "indoor", "day",
]

Challenges

Add ten more night samples. Which labels are still under-represented?
Give an example of a decision you would not want a machine to make alone, and say why.
A robot is trained only on a clean mat. What happens on a dusty one, and whose fault is that?