[RES-03-A] Circuit state machine and upstreamMeta continuity

[victornife]

Note

Sub-issue of #143. Authority: ADR 0017 (amendments 1-4).
Depends on: #283 (141-A).

Scope

The explicit CLOSED / OPEN / HALF_OPEN state machine, and the upstreamMeta change.

Transitions are guarded by a per-backend mutex; they only execute when the backend is already failing.
A single closedEpoch atomic.Uint64 is both the publication gate and the admission-generation token:
zero means "slow path required", non-zero means "CLOSED, and this is your epoch". Admission is one atomic
load; the healthy success path is closedEpoch.Load() == admissionEpoch plus fails.Load() == 0 — two
loads, no store, cheaper than today's two unconditional stores. The gate is not a duplicate of state: it
is stored zero before the state mutates on a trip and the new generation is published last when the
circuit closes, so a short state == CLOSED, gate == 0 interval is intentional. Gate and token must share
one word — a separate atomic epoch reintroduces a multi-load race, and a mutex-guarded int64 cannot be
read on a path that skips the mutex.

Ordinary failures take mu. They are exceptional, and the same critical section that increments fails
tests the threshold, so a count can never authorise a transition on its own. fails stays atomic.Int32
solely because the success fast path reads it without the lock.

Load-bearing and easy to omit: halfOpenUntil bounds HALF_OPEN itself, because a probe may be a
multi-hour gRPC stream; epoch invalidates stale results so a late probe cannot close a circuit that
has since re-opened.

circuit_half_open_probes is a real allowance — any request in the window may take a free slot — and
0 means unbounded.

Do not reintroduce a lock-free representation. Two were specified and both were wrong; see the ADR
amendments.

upstreamMeta

Remove maxFails and failTimeout. Today, changing either rebuilds the pool, restarts its health checker
and discards every backend's state. This is a behaviour change, needs a changelog entry, and will alter
existing reload tests — update them deliberately.

Call sites

Threading probe identity and epoch through result reporting touches 24 non-test call sites across 8
files — uniform t.pool.Mark{Success,Failure}(backend) shape, so mechanical, but internal/transcode/streaming.go
alone has 8.

Acceptance criteria

• One state machine; no second failure mechanism; max_fails/fail_timeout keep their names.
• Given at least N contenders, exactly circuit_half_open_probes = N admitted concurrently, proven under contention — no under-admission
  allowance.
• 0 means unbounded.
• Probe failure re-opens immediately, ignoring max_fails.
• A hung probe cannot wedge the backend.
• A stale probe result cannot close a re-opened circuit.
• Gate store order honoured: closedEpoch.Store(0) precedes the state mutation on CLOSED -> OPEN,
  and closedEpoch.Store(c.epoch) follows all state establishment on the close transition. No assertion
  anywhere that the gate mirrors state, and no bound asserted on how many requests observe a stale
  non-zero gate — the guarantee is that each of them linearizes before the trip.
• Every admission carries an epoch, ordinary requests included, and results are epoch-checked under
  mu. Test: an ordinary request admitted while CLOSED, completing after open -> recover -> close,
  neither clears live failures nor contributes to a fresh sequence.
• Ordinary failure accounting runs under mu; fails is written only under the lock. -race clean.
• maxFails/failTimeout out of upstreamMeta; tuning preserves state; changelogged.
• Active-probe recovery closes the circuit; steady-state success does not; probe failure never opens it;
  active-unhealthy suppresses probing.
• Backend state values exported for [RES-04] Resilience failure taxonomy, bounded metrics, API/Console surfaces, and soak closure #144 to project.
• L4 stream routes verified — dialBackend already drives MarkFailure/MarkSuccess, so -tags stream
  must be in the gate.

Tests

Deterministic state machine under the injected-clock seam; state-machine property test against a
model; 1000-goroutine concurrent-expiry test asserting exactly N; hung-probe re-arm; stale-epoch result
ignored; interaction with least_conn (a recovered backend has inflight == 0, so the gate must run
before the balancer); L4 breaker test. BenchmarkCircuitAdmit zero allocations.