A voice note becomes words with times
You send a voice note and an app shows what you said, with a time beside each part. The app never heard a voice. It was given a long list of numbers.
This lesson shows how sound becomes numbers, how those are cut up, and how the words come back with times.
By the end of this lesson
Note
I can cut samples into frames and join a timed transcript into lines with stamps.
Key words
| Word | What it means |
|---|---|
| Sample | one measure of how loud the sound is at one moment |
| Rate | how many samples are taken each second |
| Frame | a short run of samples treated as one piece |
| Hop | how many samples the next frame starts after the last |
| Transcript | the words that were said, with the time of each part |
The cells students fill in are left blank in this preview.
Sound is a list of samples
A microphone measures how loud the air is, again and again. Each measure is a sample, a number that can be positive or negative.
The rate says how many samples are taken each second. Speech is often 16,000 a second. A rate of 100 is used here so the lists stay short.
How long a list of samples is
| Rate (a second) | Seconds | Samples |
|---|---|---|
| 100 | 1 | 100 |
| 100 | 3 | 300 |
| 16,000 | 1 | 16,000 |
| 16,000 | 2 | 32,000 |
The cells students fill in are left blank in this preview.
Frames cut the samples into pieces
A model does not read one sample at a time. The samples are cut into frames, short runs of the same length. The hop is how far the next frame starts after the last.
With a frame of 10 and a hop of 10, frames do not overlap. With a hop of 5 they do. A start time is the frame number times the hop, divided by the rate.
A common mistake
Warning
Many people work out a frame's start time from the frame size. The next frame starts one hop later, not one frame later. If frames overlap, the two numbers differ. Always use the hop.
A transcript has times
A recogniser returns parts: some words, a start and an end in milliseconds. Parts that follow each other closely belong on one line.
Join a part to the line before it when the gap between them is less than a limit. Then write each line with a stamp such as [00:03.0].
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the sound script, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const RATE = 100;' 'const samples = [];' 'for (let i = 0; i < 80; i++) {' ' const amp = i >= 20 && i < 60 ? 40 : 2;' ' samples.push((i % 2 === 0 ? amp : -amp) + (i % 5));' '}' 'const parts = [' ' { start: 200, end: 900, text: \"turn\" },' ' { start: 950, end: 1400, text: \"the lights\" },' ' { start: 3000, end: 3600, text: \"off\" },' '];' > sound4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the sound script, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function frames(s, size, hop) {' ' const out = [];' ' for (let i = 0; i + size <= s.length; i += hop) out.push(s.slice(i, i + size));' ' return out;' '}' 'function loud(f) {' ' let t = 0;' ' for (const v of f) t += Math.abs(v);' ' return Math.floor(t / f.length);' '}' >> sound4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the sound script, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function stamp(ms) {' ' const m = Math.floor(ms / 60000);' ' const s = Math.floor((ms % 60000) / 1000);' ' const t = Math.floor((ms % 1000) / 100);' ' return \"[\" + String(m).padStart(2, \"0\") + \":\" + String(s).padStart(2, \"0\") + \".\" + t + \"]\";' '}' 'function join(list, gap) {' ' const lines = [];' ' for (const p of list) {' ' const last = lines[lines.length - 1];' ' if (last && p.start - last.end < gap) { last.text += \" \" + p.text; last.end = p.end; }' ' else lines.push({ start: p.start, end: p.end, text: p.text });' ' }' ' return lines.map(function (l) { return stamp(l.start) + \" \" + l.text; });' '}' >> sound4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the sound script, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' const fs = frames(samples, 10, 10);' ' const levels = fs.map(loud);' ' console.log(\"frames \" + fs.length + \", each \" + (10 * 1000 / RATE) + \" ms\");' ' console.log(\"loudness \" + levels.join(\" \"));' ' console.log(\"speech from frame \" + levels.findIndex(function (v) { return v >= 10; }));' ' const lines = join(parts, 300);' ' for (const l of lines) console.log(l);' ' const q = \"Write a short caption for a clip with these lines: \" + lines.join(\" \");' ' const reply = await ask([{ role: \"user\", content: q }], \"Somebody asks for the lights to go off.\", 30);' ' console.log(\"reply: \" + reply.text);' '}' 'main();' >> sound4.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Frame the sound, join the words
docker exec -w /home/node/project agent node sound4.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
The lines it prints
| Line | What it shows |
|---|---|
| frames 8, each 100 ms | 80 samples in frames of 10 at 100 a second |
| loudness 2 2 40 40 40 40 2 2 | The middle four frames are loud |
| speech from frame 2 | Frame 2 starts at 200 ms |
| [00:00.2] turn the lights | Two parts joined, as their gap was 50 ms |
| [00:03.0] off | This part was 1600 ms later, so it stands alone |
The cells students fill in are left blank in this preview.