A reading is a distribution

Mean, standard deviation, and what one number from a sensor is actually telling you.

U4.1Noise and filteringUniversity25 min

Do this lesson in the simulator

Print it: worksheet (PDF) · answers (PDF) · as a web page

Guide: Sim to real: why a policy that works in simulation fails on the robot

distance() returns 57.2. The wall is not at 57.2. The wall is somewhere near 57, and 57.2 is one draw from a distribution centred on wherever it really is.

Treating a reading as a fact is the assumption that quietly breaks every robot. Treating it as a sample is the start of everything in this module and the next three.

Two numbers describe it

For a well behaved sensor, two numbers are enough:

  • the mean, which is what it says on average, and
  • the standard deviation, sigma, which is how far a single reading usually is from that.

For a Gaussian distribution, about 68 percent of readings land within one sigma, 95 percent within two, and 99.7 percent within three. That last one is where "three sigma" comes from as an engineer's word for "practically never".

A Gaussian distribution of readings, with the share of readings within one, two and three sigma of the mean-3-2-1mean+1+2+3readings, counted in sigmas from the mean68.3% within 1 sigma95.4% within 2 sigma99.7% within 3 sigma
Where the readings of a Gaussian sensor land. The darkest band, one sigma either side of the mean, holds 68% of them; out to two sigma holds 95%; out to three, 99.7%. A reading more than three sigma out happens about 0.3% of the time.

Measuring it

Stand still, take a hundred readings, and look at them.

from bugbot import *
connect()

readings = []
for i in range(100):
    readings.append(distance())
    wait(0.1)

mean = sum(readings) / len(readings)
var = sum((r - mean) ** 2 for r in readings) / len(readings)
sigma = var ** 0.5
print("mean", round(mean, 2), "sigma", round(sigma, 2))
print("min", min(readings), "max", max(readings))
print("within 1 sigma:", sum(1 for r in readings if abs(r - mean) < sigma), "of", len(readings))

Run this in the simulator

That last line should come out near 68. When it does not, the distribution is not Gaussian and the two numbers are not enough, which is the case with outliers in U4.4.

Bias against noise, again

A sensor can be precise and wrong: small sigma, mean in the wrong place. That is bias, and no amount of averaging touches it.

  • Noise is fixed by taking more readings.
  • Bias is fixed by calibration against something you trust.

The same distinction as U3.3, and it is worth having reflexively. When a measurement is disappointing, the first question is always: is it scattered, or is it shifted?

Two faulty sensors: one scattered around the true distance, one tightly grouped around the wrong distancescattered: noisewhere the wall really isspreadfix: take more readingsand average themshifted: biaswhere the wall really isbiasfix: calibrate againstsomething you trust
Ask which one you have. On the left the readings are right on average but each one wanders; more of them cancels that out. On the right they agree closely with each other and are all wrong by the same amount, and averaging them only gives you the wrong answer more precisely.

A new reading only arrives so often

The depth sensor produces a fresh measurement about every tenth of a second. Read it twice in the same millisecond and you get the same number twice, because it is the same measurement.

That matters for averaging: a hundred reads in a tight loop is not a hundred independent samples, it is the same few samples repeated, and its mean is no better than one of them. Independence is what makes averaging work, and time is what buys independence.

The depth reading against time: it holds one value for a tenth of a second, then jumps to a new one00.10.20.30.40.50.6time, sreadingquick reads in one tenth:one measurement again and againa read every 0.1 s: a new measurement each time
The sensor makes a new measurement every tenth of a second and holds it in between. Reads a tenth apart (the dots) each get a fresh one. Quick reads inside one tenth all get the same number, so however many there are, they are worth one reading.

Task: measure the noise

Standing still, print mean: and sigma: for this sensor, in centimetres.

from bugbot import *
connect()

readings = []

Challenges

  1. Print how many readings fall within one, two and three sigma. Is it 68, 95, 99.7?
  2. Change the robot's distance from the wall and measure sigma again. Does the noise grow with range?
  3. Read the sensor a hundred times with no wait at all. What is sigma now, and why is it a lie?