Testing strategies

Black box and white box testing, alpha, beta and acceptance testing, destructive testing, and a black box test plan that finds two bugs.

A14.5Software development, law and ethicsA level45 min

At GCSE you chose normal, boundary and erroneous test data and wrote test plans (F6.3). A level adds the question of who tests, when, and with what knowledge of the code. A professional project uses several strategies, because each finds errors the others miss.

What testing is for

Testing is running the system on purpose to find errors, and to show that it meets its requirements. It looks for three kinds of error:

• syntax errors, which break the rules of the language and stop the program being translated;
• run-time errors, which crash the program while it runs, such as dividing by zero or reading past the end of a list;
• logic errors, where the program runs but gives the wrong result, such as <= where < was meant.

Testing can show that errors are present. It can never prove there are none, since no set of tests tries every possible input. So test data is chosen to be the inputs most likely to reveal errors.

Test data at A level

Boundaries matter because they are where < and <= get swapped. Any place where the expected result changes is a boundary, not just the ends of an accepted range: if speed changes at 40 cm, then 39 and 40 must both be tested.

Black box testing

In black box testing the tester does not look at the code. Tests are designed from the specification alone: for these inputs, the specification says this output. The system is a black box: inputs go in, outputs come out.

• It tests what the user actually cares about: does it do what was asked?
• The tester needs no programming knowledge, and is not misled by assumptions in the code.
• It cannot tell whether every part of the code has been run, so some paths may never be tested.

A useful technique is to divide the inputs into partitions that the specification treats the same way, then test one normal value in each partition and the boundaries between them.

White box testing

In white box (or glass box) testing the tester uses the code to design tests, aiming to run every statement, every branch of every if, and every loop zero, one and many times. It is usually done by developers, often with a trace table or the debugger.

# which branches have the tests run? record each one as it happens
ran = set()

def speed_for(gap):
    if gap < 10:
        ran.add("stop branch")
        return 0
    elif gap < 40:
        ran.add("slow branch")
        return 30
    else:
        ran.add("fast branch")
        return 60

for gap in [25, 30, 35]:          # three normal tests, all in the same partition
    speed_for(gap)
print("branches run:", sorted(ran))
print("coverage:", len(ran), "of 3")

Run this in the simulator

Three tests, but only one branch run: 2 of the 3 branches are completely untested. White box testing finds this; black box testing with careless data would not.

• It finds untested paths and dead code.
• It needs someone who can read the code, and it only tests the code that exists: it cannot notice a feature that was never written.

The two are complementary: black box checks the program does what the specification says, white box checks every part of the code has been exercised.

Who tests, and when

Acceptance testing is often in the contract: the client checks the system against the agreed success criteria, and only then signs it off and pays.

Regression testing re-runs the old tests after every change, to catch a fix that breaks something that used to work. It is why automated tests are so valuable: re-running a hundred tests by hand after every change is impractical.

Destructive testing

Destructive testing deliberately tries to break the system: impossible inputs, huge amounts of data, many users at once, the power cut mid-save, the network lost. The aim is to find out how it fails, and make sure it fails safely. For a robot: what happens if the distance sensor reads nothing, the battery dies mid-corridor, or a student stands in front of it and does not move? A robot that stops and signals for help has passed; one that pushes on has not.

Destructive tests on hardware may damage the device, which is why they are done on test units and not on the ones sold.

A test plan

A test plan is written in design, before the code, and filled in as tests are run:

The expected result comes from the specification. Filling it in after running the test turns testing into describing whatever the program happened to do.

In the NEA you test twice: iterative testing as each part is built (with evidence such as screenshots or output, and the fixes you made), and post-development or final testing of the finished program against every success criterion, ideally including feedback from real users.

Task: test the black box

A teammate has written speed_for(gap). Its specification is:

• gap is a distance in cm, and should be an int or a float.
• If gap is not a number, or is less than 0, it must raise ValueError.
• Otherwise it returns 0 if gap is less than 10, 30 if gap is from 10 up to but not including 40, and 60 if gap is 40 or more.

Do not change speed_for: your job is to test it from the specification. Make a list tests of tuples (value, expected, kind), where expected is the number the specification says, or the string "error" if it should raise ValueError, and kind is "normal", "boundary" or "erroneous". Include at least two normal tests, the four boundary values 9, 10, 39 and 40, and the erroneous values "far" and -5.

Run each test, treating a ValueError as the result "error". For each, print <value!r> <kind>: pass if the result matches, or <value!r> <kind>: FAIL (got <result>) if it does not, where <value!r> is the value as repr shows it (so the string appears as 'far'). At the end print failed: <n>. The robot stays still.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

def speed_for(gap):
    if not isinstance(gap, (int, float)):
        raise ValueError("gap must be a number")
    if gap < 10:
        return 0
    elif gap <= 40:
        return 30
    else:
        return 60

tests = []

Challenges

1. Your tests found two bugs. Fix speed_for, run the tests again, and check they all pass. What is this kind of re-testing called?
2. Write the extra white box tests needed to run every branch of the fixed function, including the one that raises the error.
3. Plan three destructive tests for the delivery robot, and say what a safe failure looks like for each.