Worksheet

A11.9 Big data

Databases and big data · A level · AQA 7517 4.11.1, Eduqas A500QS 2.5 · about 60 min

What this lesson is about

Volume, velocity and variety; why one server is not enough; distributed processing with map and reduce; the fact-based model and graph schema.

• Volume, velocity and variety
• Why one server is not enough
• Distributed processing
• The fact-based model
• Graph schema

Questions 6 marks in all

1. [1 mark]Which are the three Vs usually used to describe big data?

   Tick every answer that is true.

   1. AVolume
   2. BVelocity
   3. CVariety
   4. DValidation

2. [1 mark]Camera images, voice clips and free text arrive alongside sensor tables. Which V does this show?

   1. AVariety
   2. BVolume
   3. CVelocity
   4. DVeracity

3. [1 mark]Why does functional programming suit distributed processing of big data?

   1. AFunctions with no side effects on immutable data give the same result whichever machine runs them, in any order
   2. BFunctional programs never need more than one server
   3. CFunctional languages store data in relational tables
   4. DFunctional programs cannot fail

4. [1 mark]Which describes the fact-based model?

   1. AImmutable, timestamped facts are only ever added; the current state is worked out from them
   2. BEach fact is overwritten when it changes
   3. CFacts are stored in 3NF tables with foreign keys
   4. DOnly the latest value of each attribute is stored

5. [1 mark]In a graph schema, what does an edge represent?

   1. AA relationship between two nodes
   2. BAn entity
   3. CA property of a node
   4. DA timestamp

6. [1 mark]What does this print?

   from functools import reduce
   log = ["Ada,bump", "Bolt,bump", "Ada,bump", "Ada,stall"]
   pairs = [(line.split(",")[0], 1) for line in log if line.endswith("bump")]
   groups = {}
   for key, value in pairs:
       groups.setdefault(key, []).append(value)
   for key in sorted(groups):
       print(key, reduce(lambda a, b: a + b, groups[key]))

The task: map, shuffle, reduce

Three servers each hold part of the robots' event log. servers is a list of three lists; every item is a string "<robot>,<event>", where event is bump or stall. 1. Write mapper(line): it takes one line and returns a list of *(key, value)* pairs, [(robot, 1)] if the event is bump, or [] otherwise. 2. Call mapper on every line of every server, and shuffle: group the values by key. 3. Write reducer(key, values): it takes a robot name and the list of its values, and returns (key, total). 4. Call reducer once for each robot, in name order, and print each result as <robot> <bumps>, such as Cog 2. The robot does not move.

# the two lines every program starts with: the commands, then the robot
from bugbot import *
connect()

# each server holds part of the event log: "robot,event"
servers = [
    ["Ada,bump", "Bolt,bump", "Ada,stall"],
    ["Cog,bump", "Ada,bump", "Bolt,stall", "Bolt,bump"],
    ["Ada,bump", "Cog,bump"],
]

Plan your program here, then type it in and press Run.

Do it on the robot
www.bugbotlab.com/learn/a11-9-big-data/
The simulator checks it and tells you when it passes. Nothing to install, no account.

Challenges

1. Change the MapReduce task to find each robot's total of *all* events, and then the most common event overall. Which function changed?
2. A wrong fact says Bolt joined Hawks at time 7. Without deleting anything, how could the system correct it?
3. Draw a graph schema for a social network of students who follow each other and like robot videos.