How do robots see and move?
How robots see and move, explained simply: time-of-flight distance sensors, depth grids, cameras and bump sensors, the motors that move them, and the sense, think, act loop that joins them. Try each one on a live robot in your browser.
Robots see with sensors. A sensor measures one thing about the world, such as how far away the wall is, and turns it into a number. Robots move with motors. In between sits a program that reads the numbers and decides what the motors should do. It does this over and over, many times a second: sense, think, act.
That is the whole idea. The rest of this page shows each part on a small robot called the BugBot, in your browser. Each demo below is a real program. You can change the numbers and press Run, and the chart under the robot shows what the robot saw.
Sense, think, act
Every robot program has the same shape underneath:
repeat, many times a second:
sense read the sensors
think decide what to do
act tell the motors
wait a moment
A robot vacuum, a self-driving car and a robot arm in a factory all run a loop like this. What changes is which sensors they have, how clever the thinking is, and what the motors move. The lesson Sense, decide, act builds this loop and times it.
One thing surprises most people. A robot never sees "a wall" or "a ball". It sees numbers, such as 30 or 160. Working out what the numbers mean is the program's job.
Seeing distance: time of flight
The BugBot has a time-of-flight sensor on its front. It sends out a flash of invisible light and times how long the light takes to bounce back. Light is very fast: it travels about 30 cm in a billionth of a second. So the sensor needs a very precise clock. The longer the light takes, the further away the thing is.
In a program, distance() gives the answer in centimetres. Here the robot drives at a wall and stops when the wall is 30 cm away. The chart shows each reading.
The program
from bugbot import *
connect()
STOP_AT = 30 # stop when the wall is this close, in cm
SPEED = 40
print("wall ahead:", distance(), "cm")
while distance() > STOP_AT:
plot("distance", distance())
forward(SPEED)
wait(0.1) # look 10 times a second
stop()
for tick in range(10): # keep watching for 1 second
plot("distance", distance())
wait(0.1)
print("stopped", distance(), "cm from the wall")
The program never knows where the wall is. It keeps asking the sensor, and stops when the answer is small enough. That is what makes it a robot and not a wind-up toy.
Look at the end of the chart. The robot was told to stop at 30 but ended at 29, because it keeps sliding for a moment after the motors stop. Try SPEED = 100: it slides on to 26 cm. The lesson The distance sensor shows how to slow down as you get close.
Seeing in many directions: the depth grid
One distance only tells you about straight ahead. The BugBot's sensor measures 64 distances at once, in a grid of 8 rows by 8 columns, spread across a view 45 degrees wide. Each column looks in a slightly different direction: column 0 to the left, column 7 to the right. Rows 2 and 3 look straight out, level with the mat. The lower rows look down and see the mat itself.
Here the robot has a box ahead and to its right. The program prints the grid as a picture (# is close, . is far), then turns the robot slowly on the spot and plots three columns: the left edge, the middle and the right edge.
The program
from bugbot import *
connect()
def level(grid, col):
# the nearest thing in one column, from the two level rows
return min(grid[2 * 8 + col], grid[3 * 8 + col])
# the grid as a picture: # is close, + is medium, . is far
grid = tof_grid() # 64 distances, 8 rows of 8
for row in range(8):
line = ""
for col in range(8):
d = grid[row * 8 + col]
line += "#" if d < 40 else ("+" if d < 80 else ".")
print(line)
for tick in range(30): # 3 seconds
grid = tof_grid()
plot("left", level(grid, 0))
plot("middle", level(grid, 3))
plot("right", level(grid, 7))
turn_right(20) # spin slowly clockwise
wait(0.1)
stop()
In the picture, the box is the block of # in the top right, and the bottom four rows are the mat. As the robot turns right, the box slides across its view from right to left, the same way a lamp post slides past when you turn your head. Once the box has passed, the lines show the far edges of the mat instead.
A grid like this is how a robot finds a gap to drive through. Add up many of them as the robot moves, and it can draw a map of the room: the occupancy grid mapping guide shows how. The lesson The depth grid finds the widest gap.
Seeing with a camera
A camera gives the most information of any sensor, and it is the hardest to use. The BugBot's camera takes a picture 320 pixels wide and 240 high, which is 76,800 dots, each one just a colour. Nothing in the picture says "ball". To find a red ball, something has to look through the pixels for a patch of red, and work out where it is and how big.
On the BugBot, a small processor behind the camera does that search. Your program never gets the picture. It gets a list of what was found. With set_cv("blob", "red") the camera looks for patches of red, and blobs() returns each one as a list of numbers. The first number, cx, is how far across the picture it is: 160 is dead centre.
Here the robot finds the red ball, turns towards it and drives up to it. The chart shows two things: how many pixels the ball is from the middle of the picture, and how wide it looks.
The program
from bugbot import *
connect()
COLOUR = "red" # try "blue"
set_cv("blob", COLOUR) # camera: look for this colour
for tick in range(80):
found = blobs() # patches of the colour, largest first
if not found:
print("no", COLOUR, "in view")
break
ball = found[0] # [cx, cy, area, x0, y0, x1, y1, aspect]
off = ball[0] - 160 # pixels from the middle, + is right
width = ball[5] - ball[3]
plot("off centre", off)
plot("width", width)
if width > 40: # big in the picture means close
break
drive(40, 0, off) # forward, and turn towards it
wait(0.1)
stop()
print(COLOUR, "ball:", width, "pixels wide")
print("distance sensor:", distance(), "cm")
The camera does not know how far away the ball is. It only knows the ball looks bigger as the robot gets closer, so the program stops when the ball looks big enough. At the end the distance sensor agrees: the ball is 8 cm away. Try COLOUR = "blue" to go to the blue ball instead.
The camera can find other things too: printed markers with a number on them, faces, and lines on the floor. The lesson What the camera sees reads markers, and the line following guide uses the camera to follow a line.
Feeling: bump sensors
Sensors miss things. The depth sensor on the BugBot sits 3 cm above the mat and looks straight out, so it sees over anything very low. The camera only sees what is in front. Sooner or later a robot bumps into something it never saw, and it needs to know.
The BugBot feels a bump with its accelerometer, the same kind of sensor that tells a phone which way up it is. A collision gives it a sudden jolt, and bumped() is true for a moment afterwards. It can also notice it is stuck: a sensor underneath watches the mat go by, so if the motors are on and the mat is not moving, something is in the way. The lesson Bumps and stalls uses both.
Many robot vacuums have a bumper on the front that presses a switch when it hits something. The robot vacuum guide shows how far a robot gets with bumping alone.
How robots move
Most robots move with electric motors. On a robot with wheels, each motor turns a wheel. A robot with two wheels steers by turning one wheel faster than the other, which is how many robot vacuums get about.
The BugBot has no wheels. It has four small motors with off-centre weights on them, and when they spin they make its feet vibrate. The pattern of vibration slides it across the mat. Each foot pushes in a slightly different direction, and mixing the four lets the robot go forward, backward, sideways or spin on the spot, all without turning first.
forward(50) # drive forward at 50 percent
right(50) # slide sideways to the right, still facing forward
turn_right(30) # spin clockwise on the spot
drive(60, 60, 0) # forward, sideways and spin all at once: a diagonal
stop()
A car cannot slide sideways into a parking space. The BugBot can. A robot that can move in any direction whichever way it faces is called holonomic. Some wheeled robots manage the same trick with special wheels: the omni wheel guide shows how. The lessons Your first move and Sideways drive the BugBot.
Why robots never move perfectly
Ask a robot to drive straight and it drifts a little. No two motors are exactly the same, floors are not perfectly flat, and wheels slip. The BugBot's vibrating feet are no different: after six seconds at speed 50, the one in the simulator is facing 357.6 degrees instead of 0, and no two BugBots drift the same way. You saw another kind of error in the first demo, where the robot slid on past where it was told to stop.
A robot that just says "drive for three seconds" and never checks is called open loop. It acts and hopes. A robot that watches its sensors while it moves and corrects itself is closed loop. Every robot that has to be accurate is closed loop. That is the real reason robots have sensors: moving is never perfect, so they keep looking.
- To know where it is, a robot adds up how far it has moved and which way it turned. Small errors add up too. The dead reckoning guide shows how fast, and how a landmark fixes it.
- To correct smoothly, a robot pushes harder the further it is from where it wants to be. The PID controller guide explains how.
Putting it together: a robot that avoids things
Here is the whole loop. The robot wanders a mat with four boxes on it. Ten times a second it senses (the nearest thing in the level rows of the depth grid), thinks (is it closer than 20 cm?) and acts (drive on, or stop and turn away). The chart shows the nearest thing ahead.
The program
from bugbot import *
connect()
TOO_CLOSE = 20 # cm
turns = 0
while clock() < 18: # about 18 seconds
# SENSE: the nearest thing in the level rows of the depth grid
ahead = min(tof_grid()[16:32])
plot("nearest ahead", ahead)
# THINK: is the way ahead blocked?
if ahead < TOO_CLOSE:
# ACT: blocked, so stop and turn away
stop()
turn_right(50, angle=70)
turns += 1
else:
# ACT: clear, so drive on
forward(50)
wait(0.1)
stop()
print("turned away", turns, "times")
The gaps in the chart are the turns: the program is busy turning and does not plot. Now change the SENSE line to ahead = distance(), which looks only straight ahead. The robot clips the corner of a box with its side, because nothing in the middle of its view was close. A wider view of the world makes for a safer robot.
This is a simple robot brain, and it is enough to keep the robot out of trouble. Real robots use the same loop with more senses, more careful thinking and a map.
Questions
How do robots see?
With sensors that turn light into numbers. A distance sensor times how long a flash of light takes to bounce back, and gives a distance. A depth sensor does that in many directions at once. A camera records the colour of thousands of pixels, and a program searches them for the things it wants, such as a ball, a marker or a line. The robot never sees "a wall", only numbers that a program has to make sense of.
How do robots move?
Most use electric motors. Wheeled robots turn their wheels, and steer by running one side faster than the other. Some robots walk on legs, and some, like the BugBot, vibrate their feet to slide across the floor. A program sets how fast each motor runs.
How do robots work?
They run a loop: sense, think, act. Read the sensors, decide what to do, tell the motors, then do it all again, many times a second. Everything a robot does, from avoiding a wall to following a line, is a different version of that loop.
What is the sense, think, act cycle?
It is the loop at the heart of every robot program. Sense means reading the sensors. Think means deciding what to do with the readings. Act means sending commands to the motors. Repeating it quickly lets the robot react as the world changes. It is also called sense, plan, act, or a control loop.
What sensors do robots use?
The common ones are distance sensors (light or sound that bounces back), cameras, bump sensors, sensors that feel turning and tilting, and sensors that count how far the wheels have turned. The BugBot has a time-of-flight depth sensor, a camera, an inertial sensor that feels turns and bumps, and a sensor underneath that watches the mat go by.
How does a robot know how far away something is?
A time-of-flight sensor sends out light and times how long the reflection takes to come back. Double the distance, double the time. Ultrasonic sensors do the same with sound. A camera can also estimate distance from how big a thing of known size looks.
How does a robot camera recognise objects?
It looks for patterns in the pixels. The simplest way is colour: find the patch of pixels that is red, and that is probably the red ball. Printed markers use patterns that are easy to spot. Harder things, like faces or people, need programs that have learned what they look like from many examples.
How do robots avoid obstacles?
They check a distance sensor, a depth grid or a camera many times a second, and when something is too close they stop, turn or steer away. A bump sensor catches anything the other sensors missed. The last demo on this page is an obstacle-avoiding robot in about fifteen lines of Python.
Why do robots need sensors to move properly?
Because no motor is perfect. Wheels slip, floors are uneven and every motor is a little different, so a robot that moves without checking drifts off course. Sensors let it notice the error and correct it as it goes.
Do robots see like humans?
No. You see a room and know at once what is in it. A robot gets numbers: distances, or the colours of pixels. Everything else, like "that is a wall" or "that is a ball", has to be worked out by a program. Some robots use several cameras or depth sensors to judge distance, a little like our two eyes.
Learn it step by step
These lessons build the same ideas one at a time, each with tasks the simulator marks.
- 1.1 Your first move Driving, Robot club
- 1.4 Sideways Driving, Robot club
- 2.2 The distance sensor Sensing, Robot club
- 2.3 The depth grid Sensing, Robot club
- 4.1 What the camera sees Vision, Robot club
- 6.6 Bumps and stalls Seeing more, Robot club
- U1.1 Sense, decide, act The robot as a system, University