Hadoop lab: HDFS and MapReduce, with a file to put in and a word count to read

This lab gives each student a small Hadoop cluster of their own: HDFS, the distributed file system, with a NameNode and a DataNode, and YARN, with a ResourceManager and a NodeManager, to run MapReduce jobs. The cluster runs Hadoop’s own word count on a sample sentence as it starts, and the student puts a file of their own into HDFS, lists it, reads it back and reads the word count.

It is the Apache project’s own image, run unmodified, so there is nothing vulnerable to attack: only a system to explore. The lab copy has no route to the internet and none to any other student.

What students learn

How a big-data system stores and processes data. A file is kept by two kinds of machine: the NameNode, which knows which blocks make up which file, and the DataNode, which holds the bytes. Writing a file is two requests, the NameNode choosing a DataNode and the DataNode taking the data, and the student sees that for themselves with curl. They then read the output of a MapReduce word count, one line for each word and how many times it appeared, and see the job listed as finished in YARN. It suits data, cloud, big data and distributed systems units; check your own awarding body’s unit for the exact wording.

Run it in your browser

The lab has five services: namenode, datanode, resourcemanager, nodemanager and client. The lab’s terminal has no hadoop command, so it talks to HDFS over HTTP with curl: WebHDFS is at http://namenode:9870/webhdfs/v1/ and the list of jobs at http://resourcemanager:8088/ws/v1/cluster/apps. The cluster counted the words of a sample sentence when it started, and left the answer in /user/lab/wordcount-out; the student does not start that one. To run a MapReduce job on their own file, they use docker exec client hadoop jar in the terminal, the client service being where the hadoop command is. Nothing is saved when the lab stops, and it takes about a minute to start.

  1. Open the Container page. Open Build on the bar and choose Run a container. Anybody may open it, and the Git lab and the SQL lab start with no account. Every other lab needs a free account: create one, sign in, then press Start.
  2. Find the lab. The Sample labs card lists every sample. Find Hadoop: HDFS and MapReduce.
  3. Press Start. Start runs it as it is, in one press. Edit the file first opens the compose file so you can change it before you start.
  4. Open it. On the lab’s page the lab bar has two links, HDFS NameNode, the page that lists every file, and YARN ResourceManager, the page that lists the jobs. In the NameNode page choose Utilities, then Browse the file system, and open /user/lab. In the terminal beside the lab, curl http://namenode:9870/webhdfs/v1/user/lab/input?op=LISTSTATUS lists the same folder, and curl -L with op=OPEN on /user/lab/wordcount-out/part-r-00000 prints the word count.

What to ask students to hand in

The path of a file they put into HDFS, the lines of the word count MapReduce made from it, and a sentence on what the NameNode and the DataNode each do. Each student can choose their own text, so the answers differ. For stronger work ask why a file is read with two requests, and why a real cluster keeps three copies of every block when this one keeps one.

Nothing is marked automatically. The lab’s limits are worth saying out loud: it has one DataNode and one NodeManager, so it shows how Hadoop is arranged but not how it survives a machine failing or speeds up a big job, and permissions are off.

A basic lesson on this lab

Hadoop Basics is a small lesson worked in this lab, with each step explained and a Run button that types its command into the student’s terminal. It puts a file of one sentence into HDFS with curl, lists it and reads it back, then reads the word count the cluster made and asks the ResourceManager how the job went.

A free personal account is given it on the Basics module, which is on Home, every college can copy it from the Library’s Basics folder, and a new college opens with it in a module called Lab Basics (Sample). It is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

The code on GitHub

The compose file runs one public image five times, apache/hadoop pinned to 3.4.1 and run unmodified. Apache Hadoop is licensed under the Apache License 2.0, and we do not distribute it. Each service writes its own configuration at start, the heaps are tuned right down and the job runs as one small job, so a copy fits in the platform’s memory. The repository’s test plays a student through the lab, including a file written, read back and counted, under the same limits the platform applies.

Set it as work

Say in the task what file to put into HDFS and what to hand in.

  1. Make the lab. In your module add a task and choose the Container lab card, then In a GitHub repository. Give https://github.com/mitsurutech/lab-hadoop with the branch main. The compose file is the default, compose.yaml. Fork the repository first if you want to change it.
  2. Publish and try it. Publish pins the commit you were shown, so a later change in the repository does not alter a lab you have already set. Start your own copy first.
  3. Hand it out. Assign it like any task, with a due date if you want one, and say in the task what to do and what to hand in.
  4. Mark what is handed in. Open a student’s copy and the work they handed in, mark it, and send it back with a reason if it needs more.

Questions people ask

Why can I not read a file in the NameNode page?
The NameNode page lists the files and folders, but the bytes of a file are on the DataNode, and a browser is sent there to read them, which a lab copy does not open. Read a file with curl -L and op=OPEN in the terminal instead: the -L follows the NameNode’s redirect to the DataNode.
Are the files kept when the lab is stopped?
No. HDFS keeps its data inside the lab’s own containers, so a stopped lab starts empty again and the sample word count runs afresh. Ask students to hand in what they did before they stop it.
Does the Hadoop lab mark the word count automatically?
No. Nothing is marked automatically, so ask students to hand in the path of the file they put into HDFS and the lines of the word count made from it. A sentence on what the NameNode and the DataNode each do shows whether they understood it.