Moral, ethical, social and cultural issues

Automated decisions, AI, surveillance, personal data at scale, censorship, piracy, the environment and accessibility, with an autonomous robot that explains every decision.

A14.8Software development, law and ethicsA level45 min

Do this lesson in the simulator

At GCSE you described the impacts of technology and audited a training set for bias (F12.1 and F12.7). At A level the questions are longer and harder: you are given a situation, often one that does not exist yet, and asked to discuss the issues it raises and reach a judgement. This lesson gives you the issues the specifications name, a way to structure a discussion, and a case study: an autonomous robot that makes decisions about people.

Why computer scientists carry responsibility

AQA puts the argument plainly. Computing has transformed how information flows in society, so that it is now possible to monitor behaviour, to amass and analyse personal information, and to publish and share information about people on a scale that was never possible before. That gives the people who write the software power, and with it responsibility:

  • Algorithms embed values. Every rule in a program is a decision somebody made. A robot that gives way to adults but not to children has a value built into it, whether or not anyone meant it.
  • Scale. One program runs on millions of devices, so one developer's choice can do great good or great harm to millions of people at once.
  • Law lags behind. Technology changes faster than laws are made, and the internet crosses borders that laws stop at, so developers often face questions the law has not answered yet.

The words in the question

  • Moral (individual) issues are about what is right for a person to do: should I write code that tracks users without telling them?
  • Ethical (social) issues are about what is right for society and organisations: should companies be allowed to use face recognition in shops?
  • Legal issues are about what the law requires (lesson A14.7).
  • Cultural issues are about how technology affects, or is affected by, the beliefs, customs and ways of life of different groups.

The issues the specifications name

Computers in the workforce. Automation removes some jobs (warehouse picking, checkouts, call centres) and creates others (robot technicians, data analysts). Productivity rises, but the benefits and the losses fall on different people, and retraining takes time and money.

Automated decision making. Computers now decide, or help decide, who gets a loan, which CVs reach a human, which insurance price you pay, and where a robot drives. Automated decisions are fast, cheap and consistent, and do not get tired. But they can be biased if their rules or training data are, they can be hard to explain, people may trust them too much, and it can be unclear who is accountable when one is wrong. UK GDPR gives people rights where a decision with legal or similarly significant effects is made solely by automated means.

Artificial intelligence. Machine learning systems learn their rules from data rather than being told them, so they inherit the patterns in that data, including unfair ones. Many are opaque: even their developers cannot say exactly why one gave a particular answer. Questions follow about bias, transparency, safety, and responsibility when an AI system causes harm.

Environmental effects. Manufacturing devices uses energy and scarce materials, some mined in dangerous conditions; data centres and training large AI models use large amounts of electricity and water; discarded devices become e-waste, often shipped abroad. On the other side, computing enables smart energy grids, remote working that cuts travel, and more efficient logistics.

Censorship and the internet. Governments filter or block content, and platforms remove posts and accounts. Some of this protects people from illegal and harmful material; some suppresses dissent and free expression. What counts as acceptable differs between cultures and countries, and one global platform has to choose.

Monitoring behaviour. Employers can log keystrokes and websites, cities run CCTV with face recognition, and phones report their location. Monitoring can improve safety and catch crime, but it intrudes on privacy, can be used to control people, and changes how people behave when they know they are watched.

Analysing personal information. Big data lets organisations build detailed profiles: targeted advertising, predicting health risks, credit scoring. In 2018 it emerged that data about as many as 87 million Facebook users had been harvested and used by Cambridge Analytica for political profiling, without those users' informed consent. The same analysis that finds disease early can be used to manipulate.

Piracy and offensive communications. Digital copies are perfect and free to make, so music, films and software are easily pirated, which takes income from their creators. The ease and anonymity of online communication also allows abuse, harassment and threats, and sending grossly offensive or threatening messages can be a crime in the UK.

Layout, colour paradigms and character sets. Design choices include or exclude people. Colours carry different meanings in different cultures (red is danger in some, luck and celebration in others); some languages are written right to left; a system limited to ASCII cannot write most of the world's names, which is why Unicode matters (lesson A7). Layout and colour also decide accessibility: readable text, strong contrast, never colour alone to carry meaning, and support for screen readers.

Case study: the autonomous delivery robot

The delivery robot drives itself along busy corridors. Every line of its code is a decision about people:

  • Who does it give way to? If it stops for everyone, deliveries are slow. If it only stops for obstacles closer than 10 cm, it will bump into people who do not see it. Where to set the gap is a trade-off between efficiency and safety, and the developer is setting it for everyone in the school.
  • What does it record? A camera helps it avoid people, but recording faces creates personal data, possibly biometric data. Collect only what the decision needs, and keep it only as long as needed.
  • Is it fair? A person detector trained mostly on adults may miss small children or wheelchair users, the very people it most needs to see.
  • Who is responsible when it hurts someone? The developer who wrote the stopping rule, the school that bought it, the manufacturer of the sensor, the teacher who sent it at break time? The answer should be decided before an accident, not after.
  • Can it explain itself? A robot that logs each decision with its reason (stop, because the gap is 24 cm) can be checked, questioned and improved. A robot that just stops, or does not, cannot.

Fairness can be measured. Suppose the robot's person detector was tested in corridors, and each test records whether a person was really there and whether the robot saw them:

# (group, person was there, robot detected them)
trials = [("adult", True, True)] * 48 + [("adult", True, False)] * 2 + \
         [("child", True, True)] * 41 + [("child", True, False)] * 9 + \
         [("wheelchair user", True, True)] * 17 + [("wheelchair user", True, False)] * 3

for group in ["adult", "child", "wheelchair user"]:
    tests = [t for t in trials if t[0] == group]
    missed = sum(1 for t in tests if not t[2])
    print(f"{group}: missed {missed} of {len(tests)} ({100 * missed / len(tests):.0f}%)")

Run this in the simulator

An overall miss rate of 14 in 120 (about 12%) sounds acceptable, but it hides that the robot misses 18% of children and 15% of wheelchair users against 4% of adults. Reporting results by group, not only overall, is how bias in an automated system is found.

Structuring a discussion

Long questions on this topic are marked in levels for the quality of the argument. A structure that works:

  1. Stakeholders: who is affected? Name them from the scenario.
  2. Benefits: for whom, and why, developed with a consequence.
  3. Risks and harms: for whom, and why, developed with a consequence.
  4. Law and culture: which laws apply; would people in different cultures see it differently?
  5. Judgement: on balance, what should be done, under what conditions or safeguards?

Task: explain every decision

The robot starts 66 cm from a person standing in the corridor, facing them. Write a program that approaches safely and explains every decision it makes.

Repeat: read the gap with distance() and decide:

  • if the gap is more than 40 cm, the decision is drive: LED green, then move forward 5 cm at speed 60;
  • if the gap is more than 25 cm but not more than 40 cm, the decision is slow: LED orange, then move forward 2 cm at speed 30;
  • otherwise the decision is stop: LED red, and leave the loop without moving.

Print a line only when the decision changes from the previous one (including the first decision), in the form <time> s: <decision>, because the gap is <gap> cm, where <time> is clock() and <gap> is the reading the decision was based on. Change the LED at the same moment. After the loop, play a note and print handing over to a human. That makes exactly four lines. Do not touch the person.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

last = ""      # the previous decision, so a line is printed only when it changes

Challenges

  1. Discuss whether the school should allow the robot to use face recognition to deliver parcels to the right teacher. Use the five-step structure and reach a judgement.
  2. Change the program so the stopping gap is larger when the robot is carrying something heavy. Whose safety does this protect, and what does it cost?
  3. Give one way a delivery robot could disadvantage a wheelchair user, and one design change that prevents it.