WP-2026-005 · ml · bayesian · uncertainty

Bayesian Uncertainty Calibration and OOD Detection for Oncological Scans

Ibrahim Laklaa · 2026

experiments

OOD scores compared

-0.85

epistemic decay, log-log slope

tests on identities

Abstract

A synthetic benchmark where the Bayes posterior is known in closed form turns uncertainty quantification into a falsifiable experiment. Total uncertainty is split exactly into epistemic and aleatoric parts with MC dropout and deep ensembles, calibration is reported with bootstrap intervals, and six experiments test recovery, data scaling, temperature scaling under shift, OOD detection and deferral.

Question

In screening, a model that is wrong is survivable; a model that is wrong and confident is not. The object of study is the predictive distribution, not the prediction: where its uncertainty comes from, whether its probabilities mean what they say, and whether they support a deferral decision.

Synthetic by design, so uncertainty can be checked

On a real scan there is no ground truth for how ambiguous it is. Here a latent biomarker $z \sim N (0, 1)$ sets the label, $y \sim Bernoulli (σ (β z))$ , and is rendered as lesion contrast inside a head phantom with label-independent nuisance. The Bayes posterior $p^{*} (y ∣ x) = σ (β z)$ is known, so every uncertainty estimate has a target, and $β$ is a dial on irreducible label noise. Identifiability is bracketed rather than assumed: an analytic oracle gives a lower bound, a deliberately suboptimal matched filter an upper bound.

Decomposing uncertainty

With an approximate posterior $q (w)$ from MC dropout and a deep ensemble, total uncertainty splits exactly into epistemic and aleatoric parts:

H [E_{q} p (y ∣ x, w)] = I (y; w ∣ x) + E_{q} H [p (y ∣ x, w)]

Aleatoric noise is also learned directly, by a heteroscedastic head that puts a Gaussian on the logits, $u_{t} = f (x) + σ (x) ε_{t}$ .

Calibration done carefully

ECE is reported with 95 % bootstrap intervals and equal-mass bins, and Murphy's decomposition separates calibration from resolution.
Temperature scaling preserves accuracy for a single softmax. For an ensemble, $p_{T} = \frac{1}{M} \sum_{m} softmax (z_{m} / T)$ is a mixture and the argmax guarantee no longer holds, so the accuracy shift is measured instead of assumed.

Findings

Aleatoric estimates track the analytic target across the noise sweep, with correlation 0.74 to 0.93.
Epistemic uncertainty decays with data (log-log slope -0.85) while aleatoric stays at the target.
For novelty, Mahalanobis distance and epistemic mutual information carry the signal (AUROC up to 1.00 and 0.94 under severe noise) while softmax confidence stays near chance; aleatoric, used as a negative control, does not detect it.
A shift first labelled semantic turned out to be covariate (accuracy 0.728 vs 0.733); a decoupled condition where $p^{*} (y ∣ x) = 1/2$ exactly replaced it.
The epistemic estimate is biased downward by $O (1/ T)$ in the number of MC samples, as Jensen requires.

Rigour

44 tests mostly on mathematical identities, deterministic runs, and a NeurIPS-format report whose every number is generated from the stored results.

Read the paper Code