What you will do
Hadoop stores a very large file by cutting it into blocks and spreading them over many machines, and processes it where the blocks are. HDFS is the file system: a NameNode that knows which blocks make up which file, and DataNodes that hold them. MapReduce is the way of processing: every machine counts its own part, and the counts are added up. Your lab is a small cluster with one of each. You will put a file in, read it back and look at a word count.
Before you start
Note
Press Start the lab on the bar at the top of this lesson and wait until it says it is running. The services are namenode, datanode, resourcemanager, nodemanager and client, and they find each other by those names. Each Run button below types its command into your lab's terminal, one after another in the same session. To watch them, or to type your own, press Open terminal on the bar.
1. Put a file into HDFS
The first command writes a sentence into a file on your own terminal. The second sends it to HDFS: it asks the NameNode, which answers with the address of a DataNode, and curl follows it because of the -L and sends the bytes there. It prints nothing when it works. The folder /user/lab/mine did not exist, so HDFS made it.
echo 'one fish two fish red fish blue fish' > /tmp/fish.txtShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
curl -s -L -X PUT -T /tmp/fish.txt 'http://namenode:9870/webhdfs/v1/user/lab/mine/fish.txt?op=CREATE&user.name=lab'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
2. List it and read it back
The first command lists the folder: the answer is JSON, and for fish.txt it says "length":37 (the sentence and its line ending) and "replication":1, one copy of each block, because this cluster has one DataNode. The second reads the file back and prints the sentence. The same file is in the NameNode page in your browser, under Utilities, Browse the file system.
curl -s 'http://namenode:9870/webhdfs/v1/user/lab/mine?op=LISTSTATUS'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
curl -s -L 'http://namenode:9870/webhdfs/v1/user/lab/mine/fish.txt?op=OPEN'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
3. Read the word count
When the lab started, the cluster ran Hadoop's own word count on a sample sentence and left the answer in HDFS. The first command prints it, one line for each word and its count: the is 4, dog and fox are 2. The second asks the ResourceManager how the job went and prints "finalStatus":"SUCCEEDED". Your terminal has no hadoop command, so it cannot start a count of your own file: this lesson reads the count the lab already made.
curl -s -L 'http://namenode:9870/webhdfs/v1/user/lab/wordcount-out/part-r-00000?op=OPEN'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
curl -s http://resourcemanager:8088/ws/v1/cluster/apps | grep -o '"finalStatus":"[A-Z]*"'Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
For teachers
Tip
This lesson is a small example of a lesson worked in a lab: the steps and the Run buttons are the lesson, and the lab is each student's own copy. The lab is five services and no volumes, and nothing here marks itself: the student hands in the path of a file they put into HDFS, the lines of the word count made from it and a sentence on what the NameNode and the DataNode each do. Each student chooses their own text, so the answers differ. Copy it into your own module and change it, or build one from scratch with the Lab menu in the lesson builder.