Testing and test data

Iterative and final testing; normal, boundary, invalid and erroneous data; test plans in code.

F6.3Robust programsGCSE15 min

A program that works for the one example you tried is not a tested program. Testing is running a program on purpose with inputs chosen to find its mistakes, and checking each result against what it should be. Good testers do not try to show a program works; they try hard to make it fail, so that users never do.

When to test

• Iterative testing happens while you build: write a small piece, test it, fix it, then add the next piece. Every project in these lessons has been built this way.
• Final testing (also called terminal testing) happens when the program is finished: the whole thing is tested against its brief, to check every requirement is met.

Iterative testing finds mistakes while they are small and easy to fix. Final testing checks the pieces work together.

Choosing test data

A speed check should accept whole numbers from 0 to 100. Which values should you test it with?

Boundary data finds the most bugs, because the edges are where < and <= get mixed up.

A test plan

A test plan lists each test before it is run: what is tested, the data, and the expected result. Then the actual result is filled in:

The expected result is worked out before running the test, from the brief. If you wait to see what the program does, you will convince yourself that whatever it did was right.

Tests the program runs itself

A test plan can be turned into code: a list of test cases, and a loop that runs each one and reports pass or fail.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

def valid_speed(text):
    """True if text is a whole number from 0 to 100."""
    return text.isdigit() and 0 < int(text) < 100

tests = [("50", True), ("0", True), ("100", True), ("101", False), ("-5", False), ("fast", False)]

for number, (data, expected) in enumerate(tests, start=1):
    actual = valid_speed(data)
    result = "pass" if actual == expected else "FAIL"
    print(f"test {number}: {data!r} expected {expected}, got {actual}: {result}")

Run this in the simulator

Two tests fail: 0 and 100 are rejected. The function uses < where the brief said "from 0 to 100", which includes both ends. That is a boundary bug, exactly the kind boundary data exists to catch. Normal data alone would never have found it.

Two small new pieces of Python here. "pass" if actual == expected else "FAIL" picks one of two values in a single line. {data!r} shows a value with its quotes, so "" and "50" are clearly strings in the report.

Testing a robot

The robot's behaviour is tested the same way: decide what should happen, then check. The tasks in these lessons are automatic final tests: each one runs your program and checks it against a list of goals, such as "stopped in the green zone" or "printed exactly this line". When a task says Not yet, it is telling you which test failed.

Task: fix it with tests

The test program below has a boundary bug. Fix valid_speed so every test prints pass, without changing the tests. Then add two more tests of your own: one normal, one erroneous. All tests must pass.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

def valid_speed(text):
    """True if text is a whole number from 0 to 100."""
    return text.isdigit() and 0 < int(text) < 100

tests = [("50", True), ("0", True), ("100", True), ("101", False), ("-5", False), ("fast", False)]

for number, (data, expected) in enumerate(tests, start=1):
    actual = valid_speed(data)
    result = "pass" if actual == expected else "FAIL"
    print(f"test {number}: {data!r} expected {expected}, got {actual}: {result}")

Challenges

1. Write a test plan for the validated drive from lesson F6.1, with at least two tests of each kind of data.
2. Write valid_code(text) for six-character class codes, and a list of tests for it, before you write the function.
3. Is " 50", with a space, normal or erroneous data for valid_speed? Decide what it should do, add a test, and make it pass.