Colour, and why it breaks

Segmentation by threshold, what the threshold is really measuring, and where it fails.

U11.5VisionUniversity35 min

Print it: worksheet (PDF) · answers (PDF) · as a web page

Guide: Convolutions and CNNs explained

Colour segmentation is the first computer vision anybody writes and the first thing to fail in a demonstration. Both facts have the same cause, and it is worth understanding rather than simply being warned about.

What a camera pixel measures

A pixel is not the colour of an object. It is

pixel = illumination * reflectance * sensor response

integrated over the three channels' spectral sensitivities. The property of the object is the reflectance. The other two factors belong to the room and to the camera, and neither is constant. Move from daylight to fluorescent light and the illumination spectrum changes; the object did not.

So a threshold written against raw RGB values is a threshold against the product of three things, only one of which is the thing you care about. It works beautifully in the room where it was tuned.

Brightness is not colour

The most common beginner's test is a brightness test in disguise:

if r > 150:                 # "red"

A white sheet of paper has r well above 150. So does a pale wooden floor, and a grey wall under a warm lamp. What this test detects is "bright", with a small bias towards red.

The fix is to ask whether red dominates rather than whether it is large:

if r - max(g, b) > 60:

or, better, to divide the brightness out:

total = r + g + b + 1
if r / total > 0.5:

That second form is a chromaticity: brightness has been normalised away, and a red object in shadow and the same object in sunlight now give similar numbers. This is why real systems work in HSV or Lab rather than RGB. Hue and saturation separate the thing you want from the thing you do not.

from bugbot import *
connect()

img = camera_image(64, 48)
bright = sum(1 for row in img for (r, g, b) in row if r > 150)
chroma = sum(1 for row in img for (r, g, b) in row if r - max(g, b) > 60)
print("pixels in the picture:", 64 * 48)
print("pass 'r > 150':", bright)
print("pass 'r beats g and b':", chroma)

Run this in the simulator

The simulated camera has no lighting model at all: every surface is a flat colour and there are no shadows, no white balance and no exposure. It is the best possible case for a colour threshold. The brightness test still fails completely here, because the mat itself is nearly white and white is red-rich. If it cannot survive a world with no lighting, it will not survive a classroom with a window.

The other four ways it fails outside the lab

1. Auto exposure. The camera adjusts gain to the scene. Point it at a dark object and everything else brightens. Your thresholds were tuned against a gain that no longer applies. Lock the exposure if the driver lets you.
2. Auto white balance. The camera decides what is white and shifts the channels to suit. Walk past a red wall and it will correct your red object towards grey. Lock this too.
3. Colour constancy is a human trick, not a camera one. You perceive a banana as yellow under almost any light. The sensor does not, and its numbers move a long way.
4. Shadows and specular highlights. A shadow drops all three channels together, and that is the one change a ratio is built to survive. Dim this lesson's picture to a tenth and every pixel of the ball still passes r / total > 0.5, while r > 150 has lost the ball by half brightness. It is not a cure, though. The difference form r - max(g, b) > 60 is a brightness test in disguise and gives the ball up by a quarter brightness, and in a deep shadow there is very little left to divide: at a tenth of the light the ball reads (22, 5, 4), where the + 1 that keeps the division safe is enough to pull the red share from 0.71 to 0.69, and one count of camera noise on each channel moves it by about 0.05 either way, against 0.005 in full light. A highlight is worse than a shadow, because it saturates. It adds the same white light to all three channels, which pulls the ratio towards a third, and then the brightest channel hits 255 and the rest is cut off, and a clipped channel has thrown its information away for good.

What to do instead, in order of effort

• Calibrate in place. Show the robot the object at the start of the run and read its actual pixel values, rather than shipping constants tuned in a different room. Ten seconds of setup buys a great deal.
• Work in a brightness-invariant space. HSV, or normalised chromaticity. Cheap, and it removes the largest single failure mode.
• Use a shape or a pattern instead. A tag, a retroreflector, a known outline. This is why the fiducial markers in U11.4 exist: they are designed to be detected by structure, not by colour, and the detection is either right or absent rather than gradually wrong.
• Learn the appearance. U12.

Finding the centre

Once you have a mask, the centroid of the pixels that survive is your detection, and the column of that centroid converts to a bearing exactly as in U11.1. Remember to scale: a 64 wide picture is a downsampled 320 wide frame, so a column in the small picture is five columns in the camera's own coordinates, and the intrinsics belong to the full frame.

Task: threshold a picture

A red ball is ahead of the robot. Take a 64 by 48 picture with camera_image(64, 48) and print three things: naive:, the percentage of the picture that passes a plain r > 150 test; red:, how many pixels pass a test that looks at colour rather than brightness; and bearing:, the bearing in degrees of the centroid of those pixels.

from bugbot import *
import math
connect()

F = 92.4
W, H = 64, 48
img = camera_image(W, H)

Challenges

1. Compare your bearing with the one the built-in blob detector reports. How close, and which would you trust?
2. Redo the mask in normalised chromaticity, r / (r + g + b), and find the threshold that separates the ball from the mat. How wide is the gap between the two populations?
3. Count how many pixels the ball covers, and use that area to estimate its range as in U11.3. Compare with the detector.