Convolutions and CNNs explained
What a convolution is: a small kernel slid over a picture, what edge and blur kernels do to it, what a feature map holds, and why a CNN beats giving a network raw pixels. Run kernels over a real camera frame from a robot in your browser and see what each one finds.
A convolution is a small grid of numbers, called a kernel, slid over a picture. At every position it multiplies the pixels under it by its own numbers and adds them up, and that total becomes one pixel of a new picture. Change the nine numbers in the kernel and the same machinery finds edges, or blurs, or sharpens. A convolutional neural network, or CNN, is a network built out of kernels whose numbers are learned from examples instead of chosen by hand, and it is what sits behind nearly every program that recognises anything in an image.
On this page the robot looks at a red ball on the mat in front of it, and each demo takes that camera frame apart. Every demo is a program you can change and run.
The idea in one line
new pixel = sum of (each of the 9 pixels under the kernel x its number in the kernel)
Three things fall out of that one line, and they are the whole reason CNNs work:
- The kernel is small. Nine numbers, however big the picture.
- The kernel is used everywhere. The same nine numbers are applied at every position, so whatever they find, they find it wherever it is.
- The result is a picture, not a single number, so another kernel can be slid over it. That is what stacking layers means here.
A picture is a grid of numbers
The robot's camera returns rows of pixels, each a red, green and blue value from 0 to 255. camera_image(48, 36) asks for 48 across and 36 down. Adding the three colours and dividing by three gives brightness, one number per pixel, which is what the kernels on this page work on.
The scene is the one the robot is sitting in: a red ball about 7 cm in front of it, the mat below, the far edge of the mat as a band across the middle, and the sky above. In brightness the ball is 103, the mat is 231, the sky is 244 and the far edge is 160.
The program
from bugbot import *
connect()
CHARS = " .:-=+*#%@" # darkest to lightest
ROWS = range(14, 31) # the middle of the picture: the rest is all sky
wait(0.2)
frame = camera_image(48, 36) # 48 across, 36 down, (red, green, blue) each
grey = [[(p[0] + p[1] + p[2]) / 3 for p in row] for row in frame]
lo = min(min(row) for row in grey)
hi = max(max(row) for row in grey)
print("brightness from", round(lo), "to", round(hi))
for y in ROWS:
line = "".join(CHARS[int((v - lo) / (hi - lo) * 9.99)] for v in grey[y])
print("%2d %s" % (y, line))
print("row 23, columns 18 to 30:", [round(v) for v in grey[23][18:30]])
for x in range(48): # the brightness along one row, left to right
plot("row 23", grey[23][x])
wait(0.04)
Row 23 runs through the middle of the ball, and printed out it is 231, 231, 103, 103, 103, 103, 103, 103, 103, 103, 231, 231. Two long flat stretches with a cliff at each end. Everything a kernel does to this picture is about those cliffs.
Sliding a kernel over it
Here is the kernel this page starts with:
-1 0 +1
-2 0 +2
-1 0 +1
Put it over the picture with its middle on row 23, column 20. The nine pixels under it are the three on the left of that position, the three in the middle, and the three on the right. The left column of the kernel subtracts, the right column adds, and the middle column is ignored. So the total says how much brighter it is on the right than on the left. On flat ground the two sides are equal and the total is 0. On the ball's left edge, where 231 becomes 103, it is -512.
The program
from bugbot import *
connect()
KERNEL = [[-1, 0, 1], # change these nine numbers and press Run
[-2, 0, 2],
[-1, 0, 1]]
ROWS = range(14, 31)
def convolve(image, kernel):
"""Slide the 3 by 3 kernel over the picture: multiply each pixel under it by its
number in the kernel, add the nine up, and write the total in the middle."""
out = [[0.0] * len(image[0]) for row in image]
for y in range(1, len(image) - 1):
for x in range(1, len(image[0]) - 1):
total = 0.0
for j in range(3):
for i in range(3):
total = total + kernel[j][i] * image[y + j - 1][x + i - 1]
out[y][x] = total
return out
wait(0.2)
frame = camera_image(48, 36)
grey = [[(p[0] + p[1] + p[2]) / 3 for p in row] for row in frame]
edges = convolve(grey, KERNEL)
print("the 3 by 3 patch at row 23, column 20:")
for j in range(3):
print(" ", [round(v) for v in grey[22 + j][19:22]])
print("times the kernel, added up:", round(edges[23][20]))
for y in ROWS: # < where it gets darker to the right, > where it gets lighter
print("%2d %s" % (y, "".join("<" if v <= -200 else (">" if v >= 200 else " ") for v in edges[y])))
print("row 23, columns 18 to 30:", [round(v) for v in edges[23][18:30]])
for x in range(48):
plot("edge at row 23", edges[23][x])
wait(0.04)
The printed map is worth looking at for a moment. The ball, which was a solid blob of 103s, has become an outline: < down its left side where the picture darkens to the right, > down its right side where it lightens again, and nothing at all in the middle, because the middle is flat. The far edge of the mat, a strong horizontal line across the whole picture, produces nothing either. This kernel only sees change from left to right.
That is what a feature map is: a new picture, the same size as the old one, holding one number per position saying how strongly this kernel's pattern is present there. The kernel above is one of the Sobel pair, from 1968, and it was hand-built for exactly this job.
Two details of the arithmetic matter and are easy to miss.
The kernel needs a pixel on every side, so it cannot be centred on the outer ring of the picture. The demos leave that ring at 0. Real networks either accept a slightly smaller output or pad the picture with a ring of zeros first, which is the padding='same' you see in CNN code.
And strictly, sliding a kernel and multiplying like this is correlation. A true convolution flips the kernel top to bottom and left to right first. Every machine learning library calls this operation convolution anyway, and for a network it makes no difference, because the kernel's numbers are learned either way.
A blur kernel
Change the nine numbers and the same code does something else entirely. Nine ninths averages each pixel with its neighbours.
1/9 1/9 1/9
1/9 1/9 1/9
1/9 1/9 1/9
The program
from bugbot import *
connect()
BLUR = [[1 / 9, 1 / 9, 1 / 9], # every pixel becomes the average of its nine
[1 / 9, 1 / 9, 1 / 9],
[1 / 9, 1 / 9, 1 / 9]]
EDGE = [[-1, 0, 1],
[-2, 0, 2],
[-1, 0, 1]]
CHARS = " .:-=+*#%@"
ROWS = range(18, 30)
def convolve(image, kernel):
out = [[0.0] * len(image[0]) for row in image]
for y in range(1, len(image) - 1):
for x in range(1, len(image[0]) - 1):
total = 0.0
for j in range(3):
for i in range(3):
total = total + kernel[j][i] * image[y + j - 1][x + i - 1]
out[y][x] = total
return out
wait(0.2)
frame = camera_image(48, 36)
grey = [[(p[0] + p[1] + p[2]) / 3 for p in row] for row in frame]
blurred = convolve(grey, BLUR)
lo, hi = 103.0, 244.0
for y in ROWS:
print("%2d %s" % (y, "".join(CHARS[max(0, min(9, int((v - lo) / (hi - lo) * 9.99)))] for v in blurred[y])))
print("row 23 raw: ", [round(v) for v in grey[23][18:30]])
print("row 23 blurred:", [round(v) for v in blurred[23][18:30]])
print("edge of the raw picture: ", [round(v) for v in convolve(grey, EDGE)[23][18:30]])
print("edge of the blurred picture:", [round(v) for v in convolve(blurred, EDGE)[23][18:30]])
for x in range(1, 47): # the outer ring has no room for the kernel, so it is left out
plot("raw", grey[23][x])
plot("blurred", blurred[23][x])
wait(0.04)
The two lines on the chart show what blurring is: the raw row drops from 231 to 103 between one pixel and the next, and the blurred row takes four pixels to do it. The printed picture shows the ball with a soft rim instead of a hard one.
Blurring first and looking for edges afterwards is what nearly every real vision pipeline does, because a camera's noise is a small random change between one pixel and the next, and an edge kernel treats each of those as a tiny edge. Averaging cancels most of the noise and leaves the real cliff, which is broader than one pixel. The price is in the numbers above: the edge is found, but it is weaker and less exactly placed.
Two kernels, two feature maps
Turn the same nine numbers on their side and the kernel finds horizontal edges instead.
-1 -2 -1
0 0 0
+1 +2 +1
The program
from bugbot import *
connect()
UPRIGHT = [[-1, 0, 1], # finds edges that run up and down
[-2, 0, 2],
[-1, 0, 1]]
FLAT = [[-1, -2, -1], # finds edges that run left to right
[0, 0, 0],
[1, 2, 1]]
ROWS = range(14, 31)
COLUMN = 24
def convolve(image, kernel):
out = [[0.0] * len(image[0]) for row in image]
for y in range(1, len(image) - 1):
for x in range(1, len(image[0]) - 1):
total = 0.0
for j in range(3):
for i in range(3):
total = total + kernel[j][i] * image[y + j - 1][x + i - 1]
out[y][x] = total
return out
wait(0.2)
frame = camera_image(48, 36)
grey = [[(p[0] + p[1] + p[2]) / 3 for p in row] for row in frame]
maps = {"up and down": convolve(grey, UPRIGHT), "left to right": convolve(grey, FLAT)}
for name in maps:
print("the", name, "feature map:")
for y in ROWS:
print("%2d %s" % (y, "".join("#" if abs(v) >= 200 else ("." if abs(v) >= 60 else " ") for v in maps[name][y])))
print("column", COLUMN, "rows 16 to 28")
print(" picture: ", [round(v) for v in [grey[y][COLUMN] for y in range(16, 29)]])
for name in maps:
print(" %-14s" % name, [round(maps[name][y][COLUMN]) for y in range(16, 29)])
pixels = len(grey) * len(grey[0])
print("pixels in the picture:", pixels)
print("weights for 16 hidden neurons on the raw pixels:", pixels * 16)
print("weights for 16 kernels of 3 by 3:", 16 * 9)
for y in range(1, 35):
plot("up and down", maps["up and down"][y][COLUMN])
plot("left to right", maps["left to right"][y][COLUMN])
wait(0.05)
The two maps are different pictures of the same scene. The first outlines the ball's sides, and of the long horizontal band across the middle of the picture it finds only the two ends, where the strip of sky above it starts and stops. The second lights up along that whole band, across the ball's top and bottom, and along the sloping parts of its rim, and it has nothing at all to say on row 23, where the ball's outline runs straight up and down. Put the two together and you have the whole outline, which is more than either kernel knows on its own.
A CNN does exactly this, with more kernels. A first layer of 16 kernels gives 16 feature maps. The next layer's kernels slide over all 16 at once, so a single number in the second layer depends on a wider patch of the original picture and on several kinds of edge at once. That is how corners are built from edges, and shapes from corners.
Why a CNN beats looking at raw pixels
The network in the neural network guide takes a list of numbers and gives every input its own weight. Feed it this picture and it would need one weight per pixel: 1728 weights for a single hidden neuron, and 27,648 for a layer of 16, as the demo above prints. A convolution layer of 16 kernels needs 144.
The saving matters, but it is the smaller half of the argument. The larger half is this: a weight in an ordinary layer belongs to one pixel position. Train it on balls in the middle of the frame and it learns nothing at all about balls on the left, because those pixels have different weights. A kernel has no position. The same nine numbers are used at every position, so a kernel that responds to a round dark edge responds to one anywhere in the picture, and everything the network learns about one part of the picture it knows about all of it.
Two more parts complete a real CNN.
Pooling shrinks a feature map, usually by keeping the largest value in each 2 by 2 square. The picture gets smaller, the kernels that follow cover more of the original scene for the same nine numbers, and small shifts stop mattering.
Learning the kernels. Nobody chooses the nine numbers in a CNN. They start random and are trained exactly like any other weight, by the method in the backpropagation guide and the step in the gradient descent guide. A kernel is a set of weights that happens to be used in many places, so its share of the error is just the sum of its shares at every position it was used. Trained on photographs, the first layer usually ends up with edge kernels that look much like the ones on this page, which were worked out by hand fifty years earlier.
Where this is taught
- What the camera sees is the first look at the robot's camera and what it reports.
- Images covers pixels, resolution, colour depth and the size of a picture in bytes.
- Project: send a picture takes a frame off the robot and works on it as numbers.
- Lines uses the camera's line detector, which is built on filters like these.
- Colour, and why it breaks is about what happens to those pixel values when the light changes.
- A tiny network trains the kind of layer a CNN's kernels sit in front of.
Questions
What is a convolution in simple terms?
A small grid of numbers slid over a picture. At each position, multiply the pixels under it by the grid's numbers, add them up, and write the total into a new picture. The numbers in the grid decide what the new picture shows.
What is a kernel or filter?
The small grid of numbers. Kernel and filter mean the same thing here. Most are 3 by 3 or 5 by 5, and in a CNN one layer has many of them.
What is a feature map?
The picture that comes out of a convolution: one number per position, saying how strongly that kernel's pattern is present there. A layer with 16 kernels produces 16 feature maps from one input.
How does an edge detection kernel work?
It subtracts one side from the other. [-1, 0, 1] across a row gives zero wherever the two sides are equal, which is anywhere flat, and a large number where the brightness jumps. The Sobel kernel on this page does that on three rows at once, weighting the middle row double.
What does a blur kernel do?
It replaces each pixel with an average of the pixels around it. That removes the pixel-to-pixel randomness a camera adds, and it softens real edges at the same time, which is the trade you accept.
What is the difference between convolution and correlation?
A true convolution flips the kernel before sliding it. Correlation does not. Image code and machine learning libraries nearly always mean correlation when they say convolution, and since a network learns its kernels, the flip makes no difference to what it can learn.
What is a CNN?
A neural network whose early layers are convolutions with kernels that are learned from examples rather than chosen. The later layers are usually ordinary ones, taking what the kernels found and turning it into an answer.
Why are CNNs better than ordinary neural networks for images?
Two reasons. A kernel has nine or twenty-five weights where an ordinary layer has one per pixel, so there is far less to learn. And a kernel is used at every position, so what it learns about one part of a picture applies to every other part. An ordinary layer has to learn the same thing again for every position.
What is pooling?
Shrinking a feature map by summarising each small square, usually by taking its largest value. It makes the network cheaper, makes the layers that follow cover more of the picture, and makes small shifts of the subject matter less.
What is padding and stride?
Padding is the ring of zeros added round a picture so the kernel can be centred on the edge pixels and the output stays the same size. Stride is how far the kernel moves between positions: a stride of 2 skips every other position and halves the size of the output.
How do you write a convolution in Python?
Four nested loops: down the rows, across the columns, and then over the three by three kernel, adding up each pixel times its kernel number. That is the convolve function in the demos above, and it is what scipy.signal.convolve2d and a CNN layer do, faster.
Learn it step by step
These lessons build the same ideas one at a time, each with tasks the simulator marks.
- 4.1 What the camera sees Vision, Robot club
- F8.7 Images Data representation, GCSE
- F8.10 Project: send a picture Data representation, GCSE
- 6.1 Lines Seeing more, Robot club
- U11.5 Colour, and why it breaks Vision, University
- 8.4 A tiny network Learning, Robot club