The pinhole camera

One pixel is one ray. What that buys you, and what it does not.

U11.1VisionUniversity30 min

A depth sensor answers "how far". A camera answers "in which direction", and it answers it very precisely. Everything in this module is built on that one sentence, so it is worth being exact about what it means.

The model

Put a pinhole at the origin, looking along the optical axis, with an image plane a distance f behind it. A point in the world at (X, Y, Z) in camera coordinates, Z forward, lands on the image at

u = f * X / Z + cx
v = f * Y / Z + cy

f is the focal length in pixels, and (cx, cy) is the principal point, where the optical axis pierces the image. That is the whole of the pinhole camera. Real lenses add distortion and a finite aperture, and both are corrections to this, not replacements for it.

Notice what happened to Z. It appears only as a divisor, so doubling X and doubling Z gives the same pixel. The map from the world to the image throws depth away. A pixel does not name a point. It names a ray: the set of all world points that would land there.

What a pixel is worth

Reading it backwards, a column u corresponds to a bearing

bearing = atan((u - cx) / f)

and that is all you get from one pixel. The BugBot's camera is 320 pixels wide across a 120 degree field of view, so

f = (320 / 2) / tan(120 / 2) = 160 / tan(60) = 92.4 pixels

and the principal point sits at cx = 160, straight ahead. One pixel at the centre of the image is therefore atan(1 / 92.4), about 0.62 degrees. At the edge of the frame the same pixel is worth less, because the tangent is steeper there, which is the first reason that formula has a tangent in it and not a multiplication.

The tangent matters

It is tempting to write bearing = (u - 160) * 0.375 and be done, since 120 degrees over 320 pixels is 0.375 degrees each. Compare the two at the edge of the frame:

They agree in the middle and are 8 degrees apart a third of the way out. A robot chasing something at the edge of its view with the linear version turns the wrong amount and wonders why the loop is sluggish.

A bearing is not a position

from bugbot import *
import math
connect()

F = 92.4
set_cv("apriltag")
wait(0.3)

for tag_id, cx, cy, dist in apriltags():
    bearing = math.degrees(math.atan((cx - 160) / F))
    print("tag", tag_id, "at column", cx, "is", round(bearing, 1), "degrees off the nose")
    print("  and the detector also says", dist, "cm away")
    print("  so it is at", round(dist * math.sin(math.radians(bearing)), 1), "cm right,",
          round(dist * math.cos(math.radians(bearing)), 1), "cm ahead")

Run this in the simulator

The bearing came from the picture. The range did not: the tag detector worked it out from how large the tag looks, which is the subject of U11.3. Without a range, the bearing alone puts the tag somewhere on a line stretching off to the horizon, and no amount of image processing will tell you where along it.

This is the single most common misunderstanding about robot cameras. A detection is not a measurement of position. It is a measurement of direction, plus whatever else you can drag in: a known size, a second view, a known ground plane, a depth sensor pointed the same way.

Getting the depth back

The standard routes, in rough order of how often they are used:

• Known size. If you know how big the thing is, its apparent size gives the range. U11.3.
• A known ground plane. If the object sits on the floor and the camera height and tilt are known, the row it appears in gives the range. This is how cheap lane and obstacle systems work, and it fails the moment the floor is not flat.
• Two cameras. The same point in two images a known distance apart gives the depth by triangulation. Precision falls off as the square of the range, exactly as in U11.3.
• Motion. One camera at two times is the same triangle, if you know how far the camera moved. This is structure from motion, and it is the basis of visual odometry.

All four are the same trick. A single ray is not enough, so add a second constraint.

Task: turn a pixel into a bearing

Tag 7 is on the mat and the robot is standing still, facing along +y. Print bearing:, the angle from straight ahead to the tag in degrees, and across:, how far to the right of the robot's nose line the tag actually is in centimetres.

from bugbot import *
import math
connect()

F = 92.4
set_cv("apriltag")
wait(0.3)

Challenges

1. Work out how far wrong the linear approximation would be for this tag, in centimetres of sideways error.
2. One pixel of column error is how many centimetres of sideways error at this range? At double the range?
3. The detector reports cy as well. Given the camera sits 5 cm above the mat, what would the row of a thing standing on the floor tell you, and what would it not?