Intrinsics and calibration

Focal length in pixels, the principal point, and measuring them from known geometry.

U11.2VisionUniversity35 min

The number 92.4 in the last lesson was handed to you. On a real camera nobody hands it to you, and getting it is called calibration.

The intrinsic matrix

Everything the camera does to a ray, independent of where it is in the world, is four numbers:

K = [ fx   0   cx ]
    [  0  fy   cy ]
    [  0   0    1 ]

fx, fy: focal length in pixels, horizontally and vertically. They differ when the pixels are not square, which on a modern sensor they very nearly are, so fx is usually within a percent of fy.
cx, cy: the principal point. Nominally the centre of the image, actually a few pixels off it, because the sensor is not glued down perfectly.

Then [u, v, 1] = K * [X, Y, Z] / Z. The matrix is just the two equations from U11.1 written so that they compose with the pose of the camera.

Focal length in pixels, not millimetres

A lens is sold with a focal length in millimetres. That number alone is useless to a vision system, because what matters is how many pixels one radian covers, and that depends on the sensor as well:

f_pixels = f_mm * (image_width_pixels / sensor_width_mm)

Two cameras with the same 3.6 mm lens and different sensors have completely different intrinsics. And if you rescale an image, the intrinsics scale with it: halve the width and fx and cx both halve. Forgetting that is a classic bug, and it shows up as a system that works on full frames and mysteriously mis-aims on thumbnails.

What calibration actually measures

Calibration is not a setting. It is a least squares fit of the model to observations of known geometry.

Show the camera a target whose geometry you know exactly: usually a printed chessboard, because corners can be located to a fraction of a pixel.
Detect the target's features in a few dozen images from different angles.
Solve for the intrinsics, the lens distortion coefficients, and the unknown pose of the target in each image, all at once, by minimising the reprojection error: the distance in pixels between where each feature was seen and where the fitted model says it should have been.

The number that comes out of the process and that you should always look at is that residual. Under about 0.3 pixels of root mean square error is a good calibration. Above a pixel, something is wrong: a blurred image, a target that was not flat, or too few angles.

Different angles are the part people skip. If every image is of a chessboard square on to the camera, the focal length and the target's distance trade off against each other perfectly and the fit is degenerate. Tilt the target.

Distortion

The pinhole model is exact for a pinhole and approximate for a lens. The usual correction is radial:

r2 = x*x + y*y                          (normalised image coordinates)
x_distorted = x * (1 + k1*r2 + k2*r2*r2 + ...)

A 120 degree lens like this one is wide enough that k1 genuinely matters at the edges of the frame, often several pixels' worth. The simulated camera here is a perfect pinhole, so this module can ignore distortion, which is a simplification you should be conscious of. On real hardware you undistort first, then do the geometry.

Calibrating from what you have

You do not need a chessboard if you have something else whose geometry you know. Here you have tags at known positions and a robot at a known position, so every tag gives one correspondence between a true bearing and an observed column. The model says

u - cx = f * tan(bearing)

which is a straight line through the origin with gradient f. Fit the gradient.

from bugbot import *
import math
connect()

ROBOT = (100.0, 40.0)
TAGS = {1: (100.0, 130.0), 2: (130.0, 110.0), 3: (60.0, 90.0), 4: (140.0, 70.0)}

set_cv("apriltag")
wait(0.3)
for tag_id, cx, cy, dist in apriltags():
    tx, ty = TAGS[tag_id]
    truth = math.atan2(tx - ROBOT[0], ty - ROBOT[1])
    print("tag", tag_id, "  u - 160 =", round(cx - 160.0, 1), "  tan(bearing) =", round(math.tan(truth), 3),
          "  ratio", round((cx - 160.0) / math.tan(truth), 1) if abs(math.tan(truth)) > 0.01 else "-")

Run this in the simulator

Every ratio should land near the same number, and that is the point of a calibration: a model that fits all the observations at once, not one that was set from a single lucky measurement.

Why one point is not enough

With one correspondence you get an answer and no idea whether it is right. With four you get an answer and a residual, and the residual is what tells you the model is adequate. If the ratios disagreed by 10 percent you would know the pinhole model was not describing this lens, and go looking for distortion.

That is the real lesson of calibration and it generalises well past cameras: fit more observations than you have parameters, then look at what is left over.

Task: measure the focal length

Four tags are on the mat at (100, 130), (130, 110), (60, 90) and (140, 70). The robot stands at (100, 40) facing along +y. Print focal length:, fitted in pixels, and field of view:, the full horizontal field of view in degrees that your focal length implies.

from bugbot import *
import math
connect()

ROBOT = (100.0, 40.0)
TAGS = {1: (100.0, 130.0), 2: (130.0, 110.0), 3: (60.0, 90.0), 4: (140.0, 70.0)}
set_cv("apriltag")
wait(0.3)

Challenges

Print the residual of your fit: for each tag, the difference between the column you observed and the column your fitted f predicts. Is it under a pixel?
Leave out tag 1, whose bearing is zero. Does the fit change? Explain why that point carried no information about f.
Fit the principal point as well, by allowing an intercept in the straight line. What do you get, and is the difference from 160 meaningful given your residuals?