Four tests fail in a loaded full-suite run and pass in isolation. They are
routinely written off as flakes, but the mechanism is specific and fixable:
fixed-duration waits calibrated to a machine state rather than to the
condition being tested.
Observed
Full run 2026-08-18 on the Pi: 10 failed, 1418 passed, 1 skipped in 34:37.
CLAUDE.md documents the expected band as 10–16 min, so this run was ~2–3× more
loaded than when these constants were chosen.
Six of the ten are the session-contamination defect (#210). The other four:
| test |
failure |
test_api.py::TestRaceEndpoint::test_race_invokes_bin_px_race_for_yield_alive |
subprocess.TimeoutExpired: 'bin/px-alive --dry-run' timed out after 10 seconds |
test_gpio_lease.py::test_guard_keeps_legitimate_owner_live_then_releases |
subprocess timeout |
test_px_alive.py::test_lease_wait_does_not_touch_foreign_lease |
subprocess timeout |
test_brain.py::test_a_validated_session_is_used |
worker thread: AssertionError: no request appeared in the inbox |
All four pass in isolation.
Why this is not usefully called flakiness
A flake is nondeterministic. These are deterministic functions of load: the
timeout is a constant, the work it bounds is not. As suite runtime swings the
same code crosses the threshold and back. That produces failures that are real
signals about the harness, presented in the shape of noise — which is exactly
what trains people to skim past a red suite. The cost is already visible: it
took an explicit branch-vs-baseline comparison to establish that a merge was
safe, because "10 failed" carried no information on its own.
Note also that px-alive acquisition is documented in CLAUDE.md as normally
~6s and able to block past 15s contending for I2C — so a 10s bound is under the
known worst case even on an idle machine.
Proposed direction
- Convert fixed sleeps/timeouts to bounded condition waits (poll for the
condition with a deadline) so a slow machine waits longer rather than failing.
- Where a hard bound is genuinely needed, set it from measured worst-case with
the evidence recorded next to the constant, not from a round number.
test_brain's _pending_id helper should wait on the inbox appearing with a
deadline rather than giving up at a fixed count.
Non-goals
Four tests fail in a loaded full-suite run and pass in isolation. They are
routinely written off as flakes, but the mechanism is specific and fixable:
fixed-duration waits calibrated to a machine state rather than to the
condition being tested.
Observed
Full run 2026-08-18 on the Pi:
10 failed, 1418 passed, 1 skippedin 34:37.CLAUDE.md documents the expected band as 10–16 min, so this run was ~2–3× more
loaded than when these constants were chosen.
Six of the ten are the session-contamination defect (#210). The other four:
test_api.py::TestRaceEndpoint::test_race_invokes_bin_px_race_for_yield_alivesubprocess.TimeoutExpired: 'bin/px-alive --dry-run' timed out after 10 secondstest_gpio_lease.py::test_guard_keeps_legitimate_owner_live_then_releasestest_px_alive.py::test_lease_wait_does_not_touch_foreign_leasetest_brain.py::test_a_validated_session_is_usedAssertionError: no request appeared in the inboxAll four pass in isolation.
Why this is not usefully called flakiness
A flake is nondeterministic. These are deterministic functions of load: the
timeout is a constant, the work it bounds is not. As suite runtime swings the
same code crosses the threshold and back. That produces failures that are real
signals about the harness, presented in the shape of noise — which is exactly
what trains people to skim past a red suite. The cost is already visible: it
took an explicit branch-vs-baseline comparison to establish that a merge was
safe, because "10 failed" carried no information on its own.
Note also that
px-aliveacquisition is documented in CLAUDE.md as normally~6s and able to block past 15s contending for I2C — so a 10s bound is under the
known worst case even on an idle machine.
Proposed direction
condition with a deadline) so a slow machine waits longer rather than failing.
the evidence recorded next to the constant, not from a round number.
test_brain's_pending_idhelper should wait on the inbox appearing with adeadline rather than giving up at a fixed count.
Non-goals