Implement SmoothECE (kernel-smoothed calibration error) from Błasiok & Nakkiran 2023

[petteriTeikari]

Background

Currently using vanilla binned ECE from torchmetrics.BinaryCalibrationError in src/stats/calibration_metrics.py (line 73).

The calibration curves use LOESS smoothing for visualization, but the ECE metric itself is standard binned ECE which has known issues:

• Binning artifacts
• Inconsistent with smooth visualization
• Not principled for small sample sizes

Proposed Enhancement

Implement SmoothECE from:

Błasiok, Jarosław, and Preetum Nakkiran. 2023. "Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing." arXiv:2309.12236.

• Paper: https://arxiv.org/abs/2309.12236
• Apple implementation: https://github.com/apple/ml-calibration

Benefits

1. Principled kernel smoothing instead of arbitrary binning
2. Consistent with LOESS-smoothed calibration curves already used in figures
3. Better behavior for small sample sizes (N=208 in our case)
4. More interpretable reliability diagrams

Implementation Notes

• Reference implementation available at https://github.com/apple/ml-calibration
• Should integrate with existing src/stats/calibration_metrics.py
• Consider adding as alternative metric alongside existing ECE for comparison

Related Files

• src/stats/calibration_metrics.py - current ECE implementation
• src/r/figures/fig_calibration_dca_combined.R - calibration visualization
• docs/planning/dca-net-benefit-mini-review.md - mentioned but not implemented

Priority

Low - current LOESS visualization + Brier/IPA metrics are adequate for manuscript. This is a future enhancement for methodological rigor.

[petteriTeikari]

Status Update (2026-02-02)

Status: Not started (LOW priority as noted)

Current calibration approach in manuscript:

• Binned ECE with LOESS-smoothed calibration curves
• Brier score and IPA (scaled Brier) for overall performance
• Calibration slope and O:E ratio reported in Table S3

Recommendation: Keep open as future enhancement. Current approach is adequate for manuscript submission. SmoothECE would be nice-to-have for methodological rigor but not blocking.

---

🤖 Generated with Claude Code