The worksheetDownload the PDF
Answers

A11.9 Big data

Databases and big data · A level · AQA 7517 4.11.1, Eduqas A500QS 2.5 · about 60 min

BugBotLab

What this lesson is about

Volume, velocity and variety; why one server is not enough; distributed processing with map and reduce; the fact-based model and graph schema.

Questions 6 marks in all

  1. [1 mark]Which are the three Vs usually used to describe big data?

    Tick every answer that is true.

    1. AVolume
    2. BVelocity
    3. CVariety
    4. DValidation
    Answer: A, B, C. Too much data, arriving too fast, in too many forms.

  2. [1 mark]Camera images, voice clips and free text arrive alongside sensor tables. Which V does this show?

    1. AVariety
    2. BVolume
    3. CVelocity
    4. DVeracity
    Answer: A. Data in many forms, much of it unstructured, is variety.

  3. [1 mark]Why does functional programming suit distributed processing of big data?

    1. AFunctions with no side effects on immutable data give the same result whichever machine runs them, in any order
    2. BFunctional programs never need more than one server
    3. CFunctional languages store data in relational tables
    4. DFunctional programs cannot fail
    Answer: A. Statelessness and immutability let the work be split, run in parallel and repeated safely.

  4. [1 mark]Which describes the fact-based model?

    1. AImmutable, timestamped facts are only ever added; the current state is worked out from them
    2. BEach fact is overwritten when it changes
    3. CFacts are stored in 3NF tables with foreign keys
    4. DOnly the latest value of each attribute is stored
    Answer: A. Facts are never updated or deleted, which keeps the full history and survives mistakes.

  5. [1 mark]In a graph schema, what does an edge represent?

    1. AA relationship between two nodes
    2. BAn entity
    3. CA property of a node
    4. DA timestamp
    Answer: A. Nodes are entities, edges are the relationships between them, and properties are the data on a node.

  6. [1 mark]What does this print?

    from functools import reduce
log = ["Ada,bump", "Bolt,bump", "Ada,bump", "Ada,stall"]
pairs = [(line.split(",")[0], 1) for line in log if line.endswith("bump")]
groups = {}
for key, value in pairs:
    groups.setdefault(key, []).append(value)
for key in sorted(groups):
    print(key, reduce(lambda a, b: a + b, groups[key]))
    Answer:
    Ada 2
Bolt 1

    The map keeps only bumps as (robot, 1), the shuffle groups them, and the reduce adds each group.

The task: map, shuffle, reduce

Three servers each hold part of the robots' event log. servers is a list of three lists; every item is a string "<robot>,<event>", where event is bump or stall. 1. Write mapper(line): it takes one line and returns a list of *(key, value)* pairs, [(robot, 1)] if the event is bump, or [] otherwise. 2. Call mapper on every line of every server, and shuffle: group the values by key. 3. Write reducer(key, values): it takes a robot name and the list of its values, and returns (key, total). 4. Call reducer once for each robot, in name order, and print each result as <robot> <bumps>, such as Cog 2. The robot does not move.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

# each server holds part of the event log: "robot,event"
servers = [
    ["Ada,bump", "Bolt,bump", "Ada,stall"],
    ["Cog,bump", "Ada,bump", "Bolt,stall", "Bolt,bump"],
    ["Ada,bump", "Cog,bump"],
]

The hint students can ask for: The mapper sees one line and knows nothing else: it gives back a list of key and value pairs, empty for a line that is not a bump. The shuffle gathers every value with the same key. The reducer then sees one key and all its values.

A solution

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

# each server holds part of the event log: "robot,event"
servers = [
    ["Ada,bump", "Bolt,bump", "Ada,stall"],
    ["Cog,bump", "Ada,bump", "Bolt,stall", "Bolt,bump"],
    ["Ada,bump", "Cog,bump"],
]

def mapper(line):
    robot, event = line.split(",")
    return [(robot, 1)] if event == "bump" else []

def reducer(key, values):
    return (key, sum(values))

mapped = [pair for server in servers for line in server for pair in mapper(line)]

groups = {}
for key, value in mapped:
    groups.setdefault(key, []).append(value)

for key in sorted(groups):
    robot, total = reducer(key, groups[key])
    print(robot, total)

Any program that meets the task's checks is marked correct in the simulator; this is one way, not the only way.