Kafka lab: topics, producers, consumers, partitions and offsets in the browser

This lab gives each student their own Apache Kafka: one broker in KRaft mode, with no ZooKeeper, and Kafka UI beside it. They make a topic with three partitions, produce messages to it, read them back from the beginning and watch each partition count its own offsets, with nothing installed.

It is built from two public images and is not an attack lab, so there is nothing vulnerable in it. It is deliberately small, one broker with a 256 MB heap, for learning rather than load. The lab copy has no route to the internet and none to any other student.

What students learn

What a topic is: a named log that producers add to the end of and consumers read from wherever they choose. They see that a topic is split into partitions, that every partition keeps its own order and counts its own offsets from zero, and that reading from the beginning returns messages the reader never saw before, which is how a new consumer catches up. It suits units on event-driven architecture, messaging and queues, data pipelines and distributed systems; check your own awarding body’s unit for the exact wording.

Run it in your browser

The lab has two services: kafka, the broker, and kafka-ui, which is the page you open and also the address the lab’s terminal types to. The terminal has curl and no Kafka command-line tools, so the commands go to Kafka UI’s web API, which is what the page itself calls. The broker keeps nothing when the lab stops, so a restarted lab is an empty Kafka.

  1. Open the Container page. Open Build on the bar and choose Run a container. Anybody may open it, and the Git lab and the SQL lab start with no account. Every other lab needs a free account: create one, sign in, then press Start.
  2. Find the lab. The Sample labs card lists every sample. Find Apache Kafka: topics, producers and consumers.
  3. Press Start. Start runs it as it is, in one press. Edit the file first opens the compose file so you can change it before you start.
  4. Open it. On the lab’s page the lab bar has one link, Kafka UI. Open it, choose the cluster lab, and use Add a Topic to make a topic with three partitions. Its Produce Message button sends a message to a partition you choose, and its Overview lists each partition with its First Offset and Next Offset. In the terminal beside the lab, curl http://kafka-ui:8080/api/clusters asks the same page’s API for the cluster.

What to ask students to hand in

A screenshot of the topic’s partitions with their offsets before and after producing, and of the messages read back from the beginning. For the understanding, a sentence or two on why every partition starts counting at zero, why a message’s offset is not its position in the whole topic, and what would be different with more than one broker. For stronger work ask for a second topic with a different number of partitions.

Nothing is marked automatically. The lab’s limits are worth saying out loud: with one broker there is nothing to replicate, so replication and leader election are described, not shown.

A basic lesson on this lab

Kafka Basics is a small lesson worked in this lab, with each step explained and a Run button that types its command into the student’s terminal. It checks the cluster, makes a topic of three partitions, produces three messages to two of them, shows each partition’s offsets before and after, and reads everything back from the beginning.

A free personal account is given it on the Basics module, which is on Home, every college can copy it from the Library’s Basics folder, and a new college opens with it in a module called Lab Basics (Sample). It is deliberately small: it shows a teacher what a lesson worked in a lab is, and the lab does the rest. Copy it into a module of your own to change it.

The code on GitHub

The compose file runs two public images, each pinned to a version and run unmodified: Apache Kafka and Kafka UI (both Apache-2.0). Neither is distributed by us. The broker is told its own name on the lab’s network through its environment, and topics are never created by themselves, so a topic exists because a student made it. The repository’s test plays a student through the lab, under the same limits the platform applies.

Set it as work

Say in the task which topic to make and what to hand in.

  1. Make the lab. In your module add a task and choose the Container lab card, then In a GitHub repository. Give https://github.com/mitsurutech/lab-kafka with the branch main. The compose file is the default, compose.yaml. Fork the repository first if you want to change it.
  2. Publish and try it. Publish pins the commit you were shown, so a later change in the repository does not alter a lab you have already set. Start your own copy first.
  3. Hand it out. Assign it like any task, with a due date if you want one, and say in the task what to do and what to hand in.
  4. Mark what is handed in. Open a student’s copy and the work they handed in, mark it, and send it back with a reason if it needs more.

Questions people ask

Can a student use the Kafka command-line tools?
Not from the lab’s terminal. It has curl and no Kafka scripts, and a terminal cannot run a command inside another service. The lab teaches with Kafka UI and its web API instead, which show the same topics, partitions and offsets.
Are the messages kept when the lab is stopped?
No. The broker’s data is not saved, so stopping the lab and starting it again gives an empty Kafka. Ask students to take their screenshots before they stop.
Does the Kafka lab mark the work automatically?
No. Nothing is marked automatically, so ask for screenshots of the partitions and their offsets and of the messages read from the beginning, with a sentence on why each partition counts from zero.