← All tutorials

Black Box, Open Inquest

An audit of SENTINEL, a sepsis early-warning model, and what its vendor's explainability claim can and cannot support.

Code··········

0 / 10 questions open

A deep-learning deterioration and sepsis early-warning score running continuously over ward observations.

You are advising Brackenmoor NHS Foundation Trust. Eight weeks ago a coroner issued a Regulation 28 Prevention of Future Deaths report after a patient died of septic shock on an acute medical ward. SENTINEL, Meridian's deterioration score, had been running throughout the admission and scored her between 0.63 and 0.78 in the fourteen hours before the arrest.

Meridian's account manager has told the Trust, in writing, that 'SENTINEL is fully explainable: we supply SHAP values for every prediction.' Before the Trust replies to the coroner, or to the family's solicitor, you need to know whether that sentence is true, and what it would take to make it true.

There are ten questions across three stages. Stage A, Meridian's engineering wiki, is open now. Stage B, a public investigation of the same claim, and Stage C, the Trust's regulatory archive, stay locked until every question in the stage before them is solved. Each question is answered from the evidence in that stage.

What you should be able to do afterwards
  • Separate transparency, interpretability and explainability, and say which artefact evidences which.
  • Distinguish inherent interpretability from post-hoc explainability, and explain how model selection fixes which one you can have.
  • Choose between the xAI toolkit: feature importance, PDP, ICE, ALE, SHAP, LIME, counterfactuals, surrogate models, and the imaging methods, according to the question being asked.
  • Audit a model card against what a clinical safety case and a regulator actually require.
  • Explain how the regulatory landscape (MHRA, UK GDPR Article 22, the ICO/Turing explanation types) interacts with all three concepts.

Stage A: Wiki

Stage A is Meridian's internal engineering wiki, obtained under the disclosure order. It contains the company's own definitions of transparency, interpretability and explainability, the disclosure category SENTINEL falls into, the architecture decision record, and the explanation methods the platform supports. Using this information, you should be able to distinguish the three concepts, classify SENTINEL on the openness spectrum, and identify what the model-type decision traded away.

Open the wiki ↗

0 / 10 read

1

Question 1 of 10

≈ 12 min

Three words that are not synonyms

Before you touch anything Meridian has told the Trust, read how Meridian's own engineers describe these three ideas to each other, unpolished, on their internal wiki.

The house style note exists because someone, at some point, needed telling.

Objective

Identify the single property that supplying SHAP values on request actually evidences, and reject the plausible-sounding reasons it might seem to evidence the other two.

Meridian can supply SHAP values for any prediction 'on request'. Which statement about what that establishes is correct?

1 / 10

2

Question 2 of 10

≈ 12 min

Which category, and what follows

The licensing team keeps its own page on where Meridian's products sit on the disclosure spectrum. Read it alongside the glossary before you decide what 'commercially confidential' is actually a statement about.

Objective

Classify SENTINEL on the openness spectrum and state what that classification does and doesn't excuse.

SENTINEL publishes a model card and nothing else: no weights, no training code, no training data, and the architecture itself was only obtained under a disclosure order. Which statement is correct?

2 / 10

3

Question 3 of 10

≈ 10 min

The decision nobody revisited

One more wiki page: the architecture decision record from when SENTINEL's model type was chosen, and the engineering note on what actually went into it.

Four options were on the table. One was picked. Work out what the decision cost, and why nobody was positioned to notice.

Objective

Identify what SENTINEL's model-type decision traded away, and what it means that no formal feature selection was performed.

The architecture decision record picked Option 4; the deep transformer ensemble; over three more interpretable alternatives. Read it against the engineering note on the 340 input channels. What does this combination establish?

3 / 10

Stage locked

Stage B: Blog

An independent patient-safety blog: the coroner's report, Meridian's reply, and the blog's own tests of the model's explanations.

Unlocks after all questions in Stage A: Wiki

Stage locked

Stage C: Archive

The Trust's regulatory archive: the model card Meridian supplied, the Trust's review of it, and the correspondence with the family's solicitor.

Unlocks after all questions in Stage B: Blog