AI, robots and bias

What AI is good for, where it goes wrong, and auditing a training set for bias.

F12.7Technology and societyGCSE20 min

Covers OCR J277 1.6.1 Ethical, legal, cultural and environmental impact · AQA 8525 3.8 Ethical, legal and environmental impacts of digital technology on wider society, including issues of privacy · Edexcel 1CP2 5.2.2 AI, machine learning and robotics

Do this lesson in the simulator

Print it: worksheet (PDF) · answers (PDF) · as a web page

Machines now make decisions that used to be made by people: who gets an interview, what a camera thinks it sees, when a car brakes. Those decisions are learned from data, and data carries the marks of the world it came from. This lesson is about what AI and robots are good for, what goes wrong, and how to check a training set before trusting what it teaches.

What AI and machine learning are

Artificial intelligence is software that does things that normally need human intelligence: recognising a face, understanding speech, planning a route. Machine learning is the usual way to build it: instead of writing the rules, you show the program many examples and it finds the patterns itself. That is what module 8 did with the robot.

Because the rules are learned rather than written, nobody can simply read them, which is why these systems are hard to check.

Two rows: a person writes the rules into a program, or a program learns its rules from many exampleswriting the rulesa person writesthe rulesthe programfollows themanswersanyone can read the rulesmachine learningmanyexamplesthe programfinds the patternsanswersnobody can simply read them
Both programs give answers. The difference is where the rules come from: written rules can be read and checked, but learned rules are hidden inside the program, which is why these systems are hard to check.

Where they help

  • doing dangerous work: inspecting reactors, clearing mines, deep sea repair;
  • doing dull and repetitive work faster and more consistently than a person;
  • spotting patterns people miss, such as early signs of disease in a scan;
  • helping people: speech recognition, live captions, prosthetics, self-driving for those who cannot drive.

Where they go wrong

Bias. A model learns from its training data. If the data mostly shows one kind of person, the model works worse for everyone else: face recognition that fails on darker skin, voice recognition that struggles with regional accents, a hiring model that learned from a company's past hires and repeats its past unfairness.

No explanation. If a model refuses your loan, there may be no answer to "why?".

Responsibility. When a self-driving car hurts someone, who is accountable: the owner, the manufacturer, the programmer, the person who chose the training data? The law is still catching up.

A self-driving car hurting someone in the middle, with four people who might be accountable around ita self-driving carhurts someone:who is accountable?the ownerthe manufacturerthe programmerthe person who chosethe training datathe law is still catching up
Each of these people could be the one held responsible, so there is no single obvious answer, and the law is still working it out.

Jobs. Automation removes some jobs, changes others, and creates new ones that need different skills. The gain and the loss rarely fall on the same people.

Data hunger. These systems need huge amounts of data, often personal, which brings us back to lesson F12.2.

Auditing a training set

The first check on any model is whether its training data looks like the world it will be used in. That is just counting:

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

samples = ["day", "day", "day", "day", "day", "day", "night", "night"]
counts = {}
for s in samples:
    counts[s] = counts.get(s, 0) + 1
for name, n in counts.items():
    share = n / len(samples) * 100
    print(f"{name}: {n} ({share:.0f}%)")
print("The robot will see night far less often in training than on the mat.")

Run this in the simulator

A model trained mostly on bright daylight will do badly at dusk, and it will not warn you: it will just be confidently wrong.

A training set of six day samples and two night samples feeding a model that works in daylight and is confidently wrong at dusktraining set: 8 samplesday: 6night: 2night is only 25% of itmodelin daylight:does wellat dusk or night:confidently wrong,with no warning
The training set from the code, fed to a model. It learns daylight well because it saw so much of it, and the rarer conditions are where it fails, without telling you.

Task: audit the training set

Count how many samples of each label are in samples. Print <label>: <n> (<share>%) for each, sorted from the most common to the least, with the share rounded to the nearest whole number. Any label making up less than 15% is under-represented: add UNDER-REPRESENTED to its line. At the end print labels: <n> and under-represented: <labels>, the under-represented labels joined by ,.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

samples = [
    "day", "day", "day", "day", "day", "day", "day", "day", "day", "day",
    "day", "day", "day", "dusk", "dusk", "night", "night", "night", "indoor", "day",
]

Challenges

  1. Add ten more night samples. Which labels are still under-represented?
  2. Give an example of a decision you would not want a machine to make alone, and say why.
  3. A robot is trained only on a clean mat. What happens on a dusty one, and whose fault is that?