A million log lines before lunch
A college has 2,000 devices. Each one writes a log line every second. That is 172 million lines a day.
Some are text, some are numbers, some are pictures from cameras. Some are wrong. A spreadsheet cannot cope. This kind of problem needs new tools and new thinking.
By the end of this lesson
Note
I can name the six Vs and say how each changes the way data is kept.
Key words
| Word | What it means | Example |
|---|---|---|
| Big Data | Data so large, fast or varied that ordinary tools struggle with it. | Logs from 2,000 devices |
| Dimension | One way to measure or describe something. | Volume |
| Volume | How much data there is. | 172 million lines a day |
| Variety | How many different kinds of data there are. | Text, numbers, pictures |
| Variability | How much the data changes in shape or meaning. | The same word means different things |
| Velocity | How fast the data arrives. | One line a second per device |
| Veracity | How true and reliable the data is. | A faulty sensor gives wrong numbers |
| Value | How useful the data is for a decision. | Spotting a failing disk |
The cells students fill in are left blank in this preview.
Starter: what you already know
Have a go
Starter: what you already know
3 questionsYour answers are marked here and are not kept. A college account keeps every student’s marks.
What makes data "big"?
Big Data is data that is too large, too fast or too varied for ordinary tools. It is not only about size.
Big Data has several dimensions. There are six, and each begins with V: volume, variety, variability, velocity, veracity and value.
The six Vs and what they change
| V | Question | Effect on keeping the data |
|---|---|---|
| Volume | How much? | Needs big, cheap storage and ways to spread the work |
| Variety | How many kinds? | Needs ways to store text, numbers and pictures together |
| Variability | How much does it change? | Needs rules that cope when meaning or shape shifts |
| Velocity | How fast does it arrive? | Needs fast collection and quick processing |
| Veracity | How true is it? | Needs checks and cleaning before use |
| Value | How useful is it? | Decides if the cost of keeping it is worth paying |
The cells students fill in are left blank in this preview.
How each V changes the work
Each V changes gathering and keeping data.
High volume means more disks. High velocity means collecting and processing in real time. High variety means many formats to read. Low veracity means more time cleaning. Low value means asking if the data should be kept at all.
Variability means the same word can mean different things on different days, so the rules must be updated.
Worked example: sizing a log store
| Step | Number | Meaning |
|---|---|---|
| Devices | 100 | Each writes logs |
| Lines per device per day | 1,000 | Volume and velocity together |
| Bytes in a line | 500 | About one short sentence |
| Days kept | 10 | How long we keep them |
| Total | 500,000,000 bytes | 500 megabytes: 100 times 1,000 times 500 times 10 |
The cells students fill in are left blank in this preview.