Outliers

Some readings are not noisy, they are wrong. The median, and gating on what you expected.

U4.4Noise and filteringUniversity25 min

Do this lesson in the simulator

Noise is a reading that is a little wrong. An outlier is a reading that is not a measurement of what you think it is measuring.

Depth sensors produce them constantly: the beam misses the object and hits the far wall, or nothing at all and comes back at maximum range; something crosses in front; the surface is dark or at a sharp angle and the return is nonsense. None of that is Gaussian, and averaging it is actively harmful. One reading of 400 in an average of ten shifts the answer by 34 cm.

Watch one happen

The wall in this scene is narrow. As the robot swings, the wall leaves the sensor's view and the reading jumps to its maximum.

from bugbot import *
connect()

drive(0, 0, 40)
for i in range(40):
    print(round(distance(), 1))
    wait(0.1)
stop()

Run this in the simulator

A tidy run of numbers around 60, then 400, 400, 400, then back. A mean would be ruined by that. A median would not notice it.

The median filter

Keep the last n readings and take the middle one:

window.append(raw)
if len(window) > 5:
    window.pop(0)
estimate = sorted(window)[len(window) // 2]

A median of five survives two bad readings out of five untouched. That is what robust means in statistics: the answer does not depend on the worst data, only on the middle of it. The price is that a median rounds off genuine sharp changes too, and it costs a sort per tick, which on a microcontroller at 200 Hz is not free.

Gating

The cheaper and often better method is to refuse readings that disagree with what you expected:

if abs(raw - expected) > GATE:
    pass                 # ignore it, and count it
else:
    accept(raw)

Where does expected come from? Your model: the last estimate, or a prediction of where the wall should be given how far you have driven. This is the shape of validation gating in tracking systems, and in a Kalman filter the gate is set from the covariance so that it widens exactly when the filter is genuinely unsure.

Count what you reject. A gate that is throwing away half the readings is not protecting you, it is hiding the fact that your expectation is wrong. Print the number. A filter that silently discards data is a filter that will lie to you one day.

Task: throw out the wrong ones

Drive towards the narrow wall with a median filter running. Plot raw and median, and print thrown out:, the number of readings your gate rejected.

from bugbot import *
connect()

window = []
thrown = 0

Challenges

  1. Compare a mean of five with a median of five on the same data, plotted together.
  2. Set the gate too tight and watch the count of rejections climb. What does the estimate do?
  3. Reject any reading at maximum range before doing anything else. Is that a gate or a special case?