[RES-01-A] Resilience policy, admission primitive, and HTTP proxy integration

[victornife]

Note

Sub-issue of #141, which is the umbrella contract. Authority:
ADR 0017
(#116). Where this issue and the ADR disagree, the ADR wins.

Depends on: nothing. This is the foundation for 141-B, 141-C and 141-D, which are independent of
each other and may proceed in any order once this lands.

Scope

The resilience policy object, the admission primitive, and HTTP proxy integration. Nothing else.

Public configuration

Pool-scoped only; setting any of these in a location that targets a named upstream is a validation
error.

[upstreams.resilience]
max_active_requests    = 1000   # 0 = unlimited (default)
max_active_per_backend = 0      # 0 = unlimited (default)
max_pending_requests   = 0      # 0 = NO QUEUE, reject immediately (default)
pending_timeout        = "0s"   # 0 = bounded only by the request context

max_pending_requests = 0 means no queue, not unlimited — "unlimited pending" is the unbounded-memory
failure this exists to prevent, so it must be unrepresentable.

Deliverables

1. resilience.Policy — immutable, resolved once at load time, mirroring backendtls.Policy. Held on
   the live Pool in an atomic.Pointer. Must not enter upstreamMeta. The hot path does one pointer
   load and reads pre-parsed scalars: no tree traversal, no duration parsing, no allocation.
2. admission — in internal/upstream, a standalone type with no *Pool dependency. This is
   an import-graph commitment: internal/stream, internal/handler and later internal/auth must all
   reach it without importing a sibling.
   • atomic.Int64 CAS fast path while active < limit; no lock is taken below the limit.
   • Slow path under mu: reject if the FIFO is full, else append a waiter and select on the waiter
     channel, the request context, and pending_timeout.
   • Release with direct handoff: if a waiter exists and active-1 < limit, hand the slot to the
     FIFO head without decrementing. Strict FIFO, no barging, monotonic recovery after a limit decrease.
   • Zero new goroutines. The parked goroutine is the inbound net/http goroutine.
   • Admit returns a release closure guarded by sync.Once, capturing the exact counter it
     incremented. No generation tags.
3. proxyHandler.ServeHTTP — a new method; the type currently promotes ServeHTTP from the embedded
   *httputil.ReverseProxy. Admission is innermost, so a cache hit or a WAF-blocked request never
   consumes a slot, and background cache revalidation correctly does.
4. Per-backend limit as a selection filter, not a second queue: a backend at max_active_per_backend
   is not eligible in pickExcluding. Nested admission is a deadlock generator.
5. Lifecycle registry entries for all four paths, classified hot, plus generated mirrors. Under
   [HR-00] Make lifecycle classification granular and correct proven misclassifications #89's closed-world authority an unregistered path fails closed, so this ships here, not in 141-E.

Rejected implementations (do not reintroduce)

Counting channel (capacity is fixed at creation, and the limit is hot-reloadable);
x/sync/semaphore (no resize, unbounded waiter list); sync.Cond (cannot select on ctx.Done()); a
custom lock-free ring (no contention evidence); a timing wheel (live timers are bounded by
max_pending_requests).

Acceptance criteria

• Exactly-once acquire/release across success, transport failure, retry, cancellation, timeout, queue
  cancellation, body close, 101 upgrade and panic.
• Forced generation retirement wakes and rejects parked waiters. A parked request holds a
  handlerGen in-flight reference so the generation cannot retire gracefully, but retirement is also
  bounded by a forced grace timeout after which the transport is closed — without an explicit wakeup a
  parked request could be admitted onto a closed transport. pending_timeout is validated against the
  retirement grace.
• Reload 1000 → 100 with 500 active: the 500 complete (admission is an entry control), new arrivals
  queue or get proxy_overloaded, recovery is monotonic.
• Validation errors: stateful control in a location targeting a named upstream;
  max_pending_requests > 0 while max_active_requests = 0.
• Validation warnings: max_active_per_backend * len(servers) < max_active_requests (static lists
  only — the pool limit is then unreachable while the queue sits empty);
  max_pending_requests > 0 && pending_timeout == 0.
• Defaults reproduce current behaviour exactly.
• Lifecycle registry, generated mirrors, docs and changelog for what this slice ships.

Invariants

0 <= Q_p <= max_pending_requests
0 <= A_p <= max_active_requests + delta   (delta >= 0 only after a decrease, non-increasing)
sum(A_b) <= k * A_p                       (k = 1: a request holds pool admission during backoff
                                           and selection, but a backend slot only during an attempt)

Required tests

• Exact cap boundaries; queue full; cancel; deadline; shutdown.
• -race acquire/release/cancel against concurrent policy swaps; assert active == 0 && pending == 0
  at quiesce and delta non-increasing.
• The handoff-versus-cancel race, run repeatedly: a waiter granted a slot as it cancels must consume
  and release the grant, never leak it.
• Forced-retirement waiter wakeup.
• Reload 1000 → 100 asserted at every drain step.
• FIFO ordering and starvation freedom.

Non-goals for this slice

Connection bounds and gRPC (141-B); FastCGI/uWSGI and Backend.Network (141-C); L4 (141-D);
cross-protocol soak and benchmarks (141-E); retry (#142); circuit (#143).

Completion evidence

- PR:
- Race + handoff/cancel runs:
- Forced-retirement wakeup test:
- 1000->100 reload test:
- Lifecycle registry + mirrors:

[juliaggas]

Implementation start

• Current main SHA: 5b21472be31e42d555f391fc2dc8869e5859776b
• Governing ADR: ADR 0017 ([ADR][CGC-03] Specify generic resilience and overload-control architecture #116). Where this issue and the ADR disagree, the ADR wins.
• Parent umbrella: [RES-01] Upstream admission: concurrency, pending, and connection limits #141, under epic [EPIC][RESILIENCE] Add bounded, protocol-consistent upstream resilience #110. Programme baseline: PRE-0 on #110.
• Hard predecessors: none. [EPIC][UPSTREAM-TRUST] Enforce backend peer identity consistently #109 (backend peer trust) was reconciled and closed at this same SHA, so this slice consumes the internal/backendtls resolved policy and internal/app/generation.go staging rather than reinventing either.
• Branch: agent/issue-283-admission-foundation
• Draft PR: feat(upstream): resilience policy and admission foundation (#283) #299

Source re-audit

Verified against current main rather than assumed from the issue body:

• internal/upstream contains no resilience.go, admission.go or circuit.go. Nothing from this slice has partially landed.
• proxyHandler (internal/handler/proxy.go) does promote ServeHTTP from the embedded *httputil.ReverseProxy, exactly as the issue states, and it already implements io.Closer and is staged by internal/app/factory.go as a generation closer. The seam the issue describes is intact.
• pickExcluding exists in two places — (*PoolSnapshot).pickExcluding and (*Pool).pickExcluding, both in internal/upstream/snapshot.go — with duplicated filter bodies. The per-backend filter is factored into one selectBackend helper so the two cannot drift.
• Registry.For already resolves backendtls during the staged build and defers mutation to Commit, which is the precise seam a hot policy swap needs; the resilience policy follows the same path.

Scope

The resolved policy, the admission primitive, HTTP proxy integration, the per-backend selection filter, and the four public fields with their validation and lifecycle entries. Nothing else — connection bounds, FastCGI/uWSGI, L4, retry and the circuit all stay in their own slices.

Differences from historical issue assumptions

Three, all recorded rather than absorbed silently:

1. resilience.Policy lives in a new internal/resilience package, not inside internal/upstream. The issue names the type resilience.Policy and says it mirrors backendtls.Policy; backendtls is its own package precisely so internal/config can validate against it without an import cycle, and the resilience policy needs the same property. The admission primitive itself is in internal/upstream as specified.

2. The atomic policy pointer lives on the pool's Admission, not directly on Pool. The ADR says the live Pool holds it in an atomic.Pointer; putting the pointer on the admission the pool owns keeps one source of truth instead of two copies that could drift, and Pool.Policy()/Pool.SetPolicy() still present it as pool state. The invariant that matters — a policy change swaps a pointer and never rebuilds a pool — is unchanged.

3. The forced-retirement wakeup is generation-scoped, not pool-scoped. The ADR's state table resets the waiter FIFO on "pool close, or forced generation retirement", but a pool is reused across reload: retiring the pool's whole FIFO would reject waiters that belong to the new generation as well. Admit therefore takes the caller's retirement channel, which proxyHandler.Close closes, so exactly the waiters parked by the retiring generation are rejected. A nil channel blocks forever in select, so callers with no generation pay nothing and the fast path never touches it. Admission.Retire still exists for genuine pool close.

Verification plan

• Focused unit tests: cap boundaries, queue full, cancel, pending_timeout, FIFO ordering and starvation freedom.
• -race acquire/release/cancel against concurrent policy swaps, asserting active == 0 && pending == 0 at quiesce and a non-increasing overshoot after a decrease.
• The handoff-versus-cancel race run repeatedly under -race, asserting a grant that loses to a cancellation is consumed and released rather than leaked.
• Forced-retirement waiter wakeup through proxyHandler.Close.
• Reload 1000 → 100 with 500 active, asserted at every drain step.
• Validation errors and warnings; make generated-check for the lifecycle mirrors; make ci-fast, make test-full, scripts/docs-check.py; CI at the exact PR head for the lanes that are CI-only.