Characters: ASCII and Unicode

Character sets, bits per character, and why Unicode was needed.

F8.6Data representationGCSE12 min

Covers OCR J277 1.2.4 Data storage · AQA 8525 3.3.5 Character encoding · Edexcel 1CP2 2.2.1 Characters: 7-bit ASCII

Do this lesson in the simulator

Print it: worksheet (PDF) · answers (PDF) · as a web page

In lesson F3.2 you met character codes: A is 65. This lesson is about where those numbers come from. A character set is an agreed list of characters and the number for each, so that a message written on one computer reads the same on another. The two that matter are ASCII and Unicode.

ASCII

ASCII was agreed in the 1960s for English text. It uses 7 bits, so it has 2 to the power 7 = 128 characters: the capital and small letters, the digits, punctuation, a space, and some invisible control codes such as a new line.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for ch in ["A", "B", "Z", "a", "0", "9", " ", "!"]:
    code = ord(ch)
    print(repr(ch), code, format(code, "07b"))

Run this in the simulator

The codes are in order, so B is one after A, and a digit's code is 48 more than its value. An extended form of ASCII uses 8 bits for 256 characters, adding accented letters and symbols.

Capital letters have the codes 65, 66, 67 and so on in alphabet order; the digits 0 to 9 have the codes 48 to 57lettercodeA65B66C67D68E69……Y89Z90Z is 65 + 25digitcode048149250351……755856957the character 7 has code 55, not 7value + 48A in 7 bits: 1000001
The codes are in the same order as the alphabet and the digits, so one code gives you the rest: count on from A's 65 for a capital, or add 48 to a digit's value.

The trouble with ASCII

128 characters are enough for English, and nowhere near enough for the world. There is no room for Greek, Arabic, Hindi, Chinese, or emoji. Different countries once filled the extra codes with their own letters, so the same file could show the right text in one country and nonsense in another.

Unicode

Unicode gives a unique number to every character in every writing system: well over a hundred thousand characters, and more are added each year. Its first 128 codes are exactly ASCII, so old English text still works.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for ch in ["A", "é", "Ω", "अ", "中", "🤖"]:
    print(ch, ord(ch))

Run this in the simulator

Storage: more characters cost more bits

Every character in a file takes some bits. More possible characters means more bits for each:

  • 7-bit ASCII: 7 bits a character.
  • 8-bit extended ASCII: 1 byte a character.
  • Unicode, in the common UTF-8 encoding: 1 byte for ASCII characters, and 2, 3 or 4 bytes for everything else.
# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

for message in ["robot", "robôt", "机器人", "🤖🤖"]:
    utf8 = message.encode("utf-8")
    print(message, len(message), "characters,", len(utf8), "bytes in UTF-8")

Run this in the simulator

encode("utf-8") turns text into the bytes that are really stored or sent. A robot sending messages by radio pays for every byte, which is one reason simple messages use plain ASCII.

In UTF-8, r, o, b and t take one byte each, ô takes two and each of the three Chinese characters takes threerobotrobot5 characters, 5 bytesrobôtrobôt5 characters, 6 bytes机器人机器人3 characters, 9 byteseach small bar is one byte
Every character that is also in ASCII still takes one byte, so English text costs no more in UTF-8. Only the characters outside ASCII take two, three or four bytes.

Working out text size

The size of some text is the number of characters times the bits per character:

"BugBot ready" is 12 characters.
In 7-bit ASCII: 12 × 7 = 84 bits.
In 8-bit ASCII: 12 × 8 = 96 bits = 12 bytes.

Task: message sizes

For the message BugBot says hi, print three lines: characters: <n>, ascii bits: <n> for 7-bit ASCII, and utf-8 bytes: <n>. Then print the 7-bit binary code of each of its first three characters, one per line, such as B 1000010. Work everything out from the message.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

message = "BugBot says hi"

Challenges

  1. Find the smallest code for a lower-case letter, and explain how to turn any capital into lower case with arithmetic.
  2. Send the robot's name as a string of 7-bit binary codes, and decode it again.
  3. Which uses fewer bytes in UTF-8: a tweet in English or the same length of emoji? Measure it.