The Price of the Model
An audit of VITAL-LM, a clinical language model, covering re-identification risk, the cost of training it, and the carbon of running it.
0 / 10 questions open
A clinical language model that drafts discharge summaries and suggests diagnostic codes from free-text notes.
You sit on Brackenmoor NHS Foundation Trust's Information Governance and Digital Investment Committee. VITAL-LM's contract renews in six weeks, and Meridian is also pitching a five-trust expansion. The board paper in front of you, and the marketing site behind it, promises a return of nearly £900,000 a year, 'zero privacy risk because the data is de-identified', and an environmental impact that is 'negligible'.
There are ten questions across three stages. Stage A, Meridian's marketing microsite, is open now. Stage B and Stage C stay locked until every question in the stage before them is solved. Each question is answered from the evidence in that stage.
The three claims concern privacy, cost and the environment. Each is defensible in principle and each, in this dossier, is wrong in a different way. Bring a calculator.
What you should be able to do afterwards
- Explain why centralising patient data is risky, and compute k-anonymity over a quasi-identifier set.
- Distinguish centralised, distributed, federated and split learning, and choose between weighted and unweighted federated averaging.
- Explain differential privacy and the ε parameter, and identify which control acts on memorisation at its root.
- Account for what drives the compute cost of a model: size, data, iterations and hyperparameter search, and evaluate early stopping, transfer learning, quantisation, pruning, distillation and cascades.
- Estimate the operational carbon of a deployed model, compare it honestly against training, and separate cashable savings from time released to care.
Stage A: Brochure
Stage A is Meridian's marketing microsite for VITAL-LM. It contains the product claims, the savings calculator inputs, the privacy page, and a sample of the de-identified data the company publishes as a transparency exhibit. Using this information, you should be able to test the anonymity of the extract, check the savings arithmetic, and identify which of the site's claims survive scrutiny.
0 / 11 read
Question 1 of 10
≈ 12 min
'De-identified' is a claim, not a property
The privacy page says there is no privacy risk because the extract is de-identified, and publishes ten sample rows 'so you can check our working'. Take it at its word.
Treat sex, year of birth and postcode district as the quasi-identifier set: the attributes an adversary could plausibly know or obtain from an electoral roll, a social media profile or a local news report.
Objective
Measure how anonymous this extract actually is.
1 / 10
Question 2 of 10
≈ 12 min
Check the working
Before you argue with Meridian's £900,000 headline, check whether it's even arithmetically honest. Unusually for a vendor, they've published the four inputs behind it.
Objective
Verify Meridian's own savings arithmetic, to the nearest £1,000.
2 / 10
Question 3 of 10
≈ 10 min
Read the small print of the big claims
You've now checked one claim carefully. Before you leave the microsite, audit the rest of it the same way, line by line, against what the pages actually say rather than what they imply.
Objective
Identify which claims on the site survive scrutiny and which don't.
3 / 10
Stage locked
Stage B: Service Desk
The Trust's service desk: the tickets the Data Protection Officer and data engineering team raised, a briefing on training architectures, and the quarterly retrain log.
Unlocks after all questions in Stage A: Brochure
Stage locked
Stage C: Committee Papers
The committee papers: the red-team log, the differential privacy briefing, the compute telemetry, and the draft determination.
Unlocks after all questions in Stage B: Service Desk