Keeping data, then deleting it
Data retention is the rule that says how long an app keeps each kind of data. After that time, the data is deleted.
It matters because a chat app that keeps every conversation for ever holds more risk each year. Here you sweep old data, delete on request and count what is left.
By the end of this lesson
Note
I can delete data on a rule and on a person's request.
Key words
| Word | What it means | Example |
|---|---|---|
| Retention | how long data is kept | |
| Sweep | a job that deletes what is too old | |
| Deletion request | a person asking for their data to be removed | |
| Copy | another place the same data is stored | a cache |
| Backup | a saved copy kept to recover from a fault |
The cells students fill in are left blank in this preview.
A retention rule sets a period
Different data needs different times. A chat may be kept for 30 days. A bill is kept longer, because it is needed for accounts. This lesson uses 365 days.
Each record has a day it was made. A sweep works out its age, today minus that day. If the age is more than the period, the record goes. A record at exactly the limit stays.
Worked example: a sweep on day 100
| Record | Kind | Made on day | Age | Period | Result |
|---|---|---|---|---|---|
| c1 | chat | 10 | 90 | 30 | Deleted |
| c2 | chat | 80 | 20 | 30 | Kept |
| b1 | billing | 20 | 80 | 365 | Kept |
| c3 | chat | 70 | 30 | 30 | Kept, exactly at the limit |
The cells students fill in are left blank in this preview.
A deletion request must reach every copy
A person may ask you to delete their data. Their messages may live in the main store, in a cache that speeds things up and in a backup.
A deletion that reaches only the main store leaves the rest. List every place first. In UK and EU rules this is the right to erasure, which has limits, so ask an expert. This is not legal advice.
A common mistake
Warning
Many people think deleting from the main store deletes the person's data. It does not. Caches, logs and backups keep copies. A backup is usually not edited, so it clears when it expires. Tell the person this, with the date.
Write the model helper, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const base = process.env.AI_URL || \"http://ai:4000\";' '// The token counts of a reply, or null when the reply carries none.' 'function count(data, name) {' ' return data.usage && typeof data.usage[name] === \"number\" ? data.usage[name] : null;' '}' > model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the model helper, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function ask(messages, sample, max) {' ' try {' ' const res = await fetch(base + \"/v1/chat/completions\", {' ' method: \"POST\",' ' headers: { \"content-type\": \"application/json\", authorization: \"Bearer lab\" },' ' body: JSON.stringify({ model: \"lab\", messages: messages, max_tokens: max || 100 }),' ' signal: AbortSignal.timeout(30000),' ' });' ' if (res.ok === false) throw new Error(\"status \" + res.status);' ' const data = await res.json();' ' const text = data.choices[0].message.content;' ' return { text, inTokens: count(data, \"prompt_tokens\"), outTokens: count(data, \"completion_tokens\"), recorded: false };' ' } catch (error) {' ' console.log(\"Recorded sample: no model answered (\" + error.message + \").\");' ' return { text: sample, inTokens: null, outTokens: null, recorded: true };' ' }' '}' 'module.exports = { ask };' >> model.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retention sweep, part 1
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'const { ask } = require(\"./model\");' 'const rules = { chat: 30, billing: 365 };' 'const today = 100;' 'const records = [' ' { id: \"c1\", user: \"u1\", day: 10, kind: \"chat\" },' ' { id: \"c2\", user: \"u2\", day: 80, kind: \"chat\" },' ' { id: \"b1\", user: \"u1\", day: 20, kind: \"billing\" },' ' { id: \"c3\", user: \"u1\", day: 70, kind: \"chat\" },' '];' > retain.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retention sweep, part 2
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function expired(r) {' ' return typeof rules[r.kind] === \"number\" && today - r.day > rules[r.kind];' '}' 'function ids(list) {' ' return list.map(function (r) { return r.id; });' '}' 'const stores = {' ' live: records.filter(function (r) { return expired(r) === false; }),' ' cache: [records[3]],' ' backup: [records[0], records[2], records[3]],' '};' >> retain.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retention sweep, part 3
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'function remaining(user, reached) {' ' const left = {};' ' Object.keys(stores).forEach(function (name) {' ' const mine = stores[name].filter(function (r) { return r.user === user; });' ' if (reached.indexOf(name) < 0 && mine.length > 0) left[name] = ids(mine);' ' });' ' return left;' '}' >> retain.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Write the retention sweep, part 4
docker exec -w /home/node/project agent sh -c "printf '%s\n' 'async function main() {' ' console.log(\"expired today: \" + JSON.stringify(ids(records.filter(expired))));' ' console.log(\"kept by sweep: \" + JSON.stringify(ids(stores.live)));' ' console.log(\"delete u1 from live and cache, left: \" + JSON.stringify(remaining(\"u1\", [\"live\", \"cache\"])));' ' console.log(\"delete u1 from all three, left: \" + JSON.stringify(remaining(\"u1\", [\"live\", \"cache\", \"backup\"])));' ' console.log(\"backup made on day 90, kept 35 days, days left: \" + Math.max(0, 90 + 35 - today));' ' const q = \"In one short sentence, why can a backup still hold data after it was deleted?\";' ' const sample = \"A backup is a separate copy, so it only changes when it expires.\";' ' const reply = await ask([{ role: \"user\", content: q }], sample, 60);' ' console.log(\"model said: \" + reply.text);' '}' 'main();' >> retain.js"Shown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
Run the retention sweep
docker exec -w /home/node/project agent node retain.jsShown here, not run. Create a free account to press Run and type it into your own lab’s terminal.
What the Run prints
| Line | What it shows |
|---|---|
| expired today: ["c1"] | Only c1 is older than its period |
| kept by sweep: ["c2","b1","c3"] | What stays in the main store |
| delete u1 from live and cache, left: {"backup":[...]} | The backup still holds u1 |
| delete u1 from all three, left: {} | Every copy reached |
| days left: 25 | When the backup clears |
| model said: ... | A sentence. The words differ each run |
The cells students fill in are left blank in this preview.