Characters: ASCII and Unicode

Character sets, bits per character, and why Unicode was needed.

F8.6Data representationGCSE12 min

Do this lesson in the simulator

In lesson F3.2 you met character codes: A is 65. This lesson is about where those numbers come from. A character set is an agreed list of characters and the number for each, so that a message written on one computer reads the same on another. The two that matter are ASCII and Unicode.

ASCII

ASCII was agreed in the 1960s for English text. It uses 7 bits, so it has 2 to the power 7 = 128 characters: the capital and small letters, the digits, punctuation, a space, and some invisible control codes such as a new line.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for ch in ["A", "B", "Z", "a", "0", "9", " ", "!"]:
    code = ord(ch)
    print(repr(ch), code, format(code, "07b"))

Run this in the simulator

The codes are in order, so B is one after A, and a digit's code is 48 more than its value. An extended form of ASCII uses 8 bits for 256 characters, adding accented letters and symbols.

The trouble with ASCII

128 characters are enough for English, and nowhere near enough for the world. There is no room for Greek, Arabic, Hindi, Chinese, or emoji. Different countries once filled the extra codes with their own letters, so the same file could show the right text in one country and nonsense in another.

Unicode

Unicode gives a unique number to every character in every writing system: well over a hundred thousand characters, and more are added each year. Its first 128 codes are exactly ASCII, so old English text still works.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for ch in ["A", "é", "Ω", "अ", "中", "🤖"]:
    print(ch, ord(ch))

Run this in the simulator

Storage: more characters cost more bits

Every character in a file takes some bits. More possible characters means more bits for each:

  • 7-bit ASCII: 7 bits a character.
  • 8-bit extended ASCII: 1 byte a character.
  • Unicode, in the common UTF-8 encoding: 1 byte for ASCII characters, and 2, 3 or 4 bytes for everything else.
# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for message in ["robot", "robôt", "机器人", "🤖🤖"]:
    utf8 = message.encode("utf-8")
    print(message, len(message), "characters,", len(utf8), "bytes in UTF-8")

Run this in the simulator

encode("utf-8") turns text into the bytes that are really stored or sent. A robot sending messages by radio pays for every byte, which is one reason simple messages use plain ASCII.

Working out text size

The size of some text is the number of characters times the bits per character:

"BugBot ready" is 12 characters.
In 7-bit ASCII: 12 × 7 = 84 bits.
In 8-bit ASCII: 12 × 8 = 96 bits = 12 bytes.

Task: message sizes

For the message BugBot says hi, print three lines: characters: <n>, ascii bits: <n> for 7-bit ASCII, and utf-8 bytes: <n>. Then print the 7-bit binary code of each of its first three characters, one per line, such as B 1000010. Work everything out from the message.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

message = "BugBot says hi"

Challenges

  1. Find the smallest code for a lower-case letter, and explain how to turn any capital into lower case with arithmetic.
  2. Send the robot's name as a string of 7-bit binary codes, and decode it again.
  3. Which uses fewer bytes in UTF-8: a tweet in English or the same length of emoji? Measure it.