[RES-01-B] Connection bounds and HTTP/2, HTTP/3, WebSocket and gRPC lifetime accounting

[victornife]

Note

Sub-issue of #141. Authority:
ADR 0017
(#116).

Depends on: #283 (141-A) only. Independent of 141-C and 141-D — these three may proceed in parallel.

Scope

Physical connection bounds, plus admission integration and lifetime accounting for every remaining
in-tree protocol.

Public configuration

Stateless, so settable at pool or location level (location wins) — transports are already built per
location by newProxyTransport(loc, policy).

max_connections_per_backend = 0   # 0 = unlimited; HTTP/1.1 effective

The name is a one-way door and was decided deliberately. max_connections was rejected: Go can only
enforce per host per transport, and [rate_limit] max_conns already denotes an inbound per-listener
cap. Two similar names pointing in opposite directions is the trap ADR 0016 avoided by keeping tls and
backend_tls distinct.

Deliverables

1. max_connections_per_backend → http.Transport.MaxConnsPerHost — the only lever Go offers that
   bounds sockets without defeating pooling and that honours the request context while queueing for a dial.
   • Not applicable to native gRPC or transcoding, where one HTTP/2 connection carries all streams;
     validation warns when set on such a route.
   • Health-check clients are exempt, so a saturated pool can still observe recovery.
   • Idle connections count toward the limit until IdleConnTimeout.
2. Native gRPC and transcoding admission — internal/handler/grpcproxy.go and internal/transcode
   have their own handler entry points and each needs the Admit call from 141-A.
3. Lifetime accounting verified across protocols, using the existing release discipline rather than
   new mechanisms.

Accounting matrix (normative)

Client protocol	A_p / A_b	Backend C_b	Which limit binds
HTTP/1.1, no keep-alive	+1 per request	+1 per request	both, in lockstep
HTTP/1.1 keep-alive	+1 per request	conns reused; idle conns count until IdleConnTimeout	either; C_b can bind while A_b is low
HTTP/2	+1 per stream	typically 1	request limit; connection limit rarely binds
HTTP/3	+1 per stream	unaffected — the backend leg is still HTTP/1.1 or h2	request limit only
WebSocket / 101	+1 until releaseRWBody.Close()	+1 dedicated, hijacked, never pooled	both
Server-sent events	+1 for the response lifetime	as HTTP/1.1	request limit
gRPC unary	+1 per call	shared h2 conn	request limit
gRPC server/client/bidi streaming	+1 for the whole stream lifetime	shared	request limit
gRPC transcoding, unary and streaming	+1 per call/stream	shared	request limit

HTTP/3 requires no protocol-specific code. Jul's HTTP/3 support is inbound only — internal/server/http3.go
runs a quic-go http3.Server, and there is no HTTP/3 backend transport. HTTP/3 requests are served by the
same handler tree via acquireGen() and upstream.WithSnapshot, so admission applies unchanged.
h3Conn.Close(ctx) does GOAWAY and drain bounded by shutdown_timeout, so a long-lived h3 stream holds
its slot and its handlerGen reference exactly as an h2 stream does. WebSocket over HTTP/3 is not
applicable — h3 uses extended CONNECT, which Jul does not implement.

The operator-facing sentence for the docs: max_active_requests counts requests and streams;
max_connections_per_backend counts sockets. Under HTTP/2, HTTP/3 and gRPC one socket carries many
streams, so the request limit normally binds first; under HTTP/1.1 and WebSocket the two move together.

Acceptance criteria

• MaxConnsPerHost wired from the resolved policy; warns on gRPC and transcoding routes; health-check
  clients exempt.
• Native gRPC and transcoding acquire and release admission on every terminal path, including
  half-open and cancelled streams.
• Every row of the accounting matrix is asserted by a test.
• A long-lived stream holds its slot for its full lifetime and releases exactly once.
• Lifecycle registry entry for max_connections_per_backend, docs for the accounting matrix.

Required tests

• Per-protocol accounting: HTTP/1.1 keep-alive, HTTP/2 multi-stream, HTTP/3, WebSocket lifetime, SSE,
  gRPC unary and all three streaming shapes, transcoding.
• Client disconnect and panic release the slot on each protocol.
• A MaxConnsPerHost integration test gating the documented semantics, covering interaction with
  MaxIdleConnsPerHost: 32 under keep-alive churn. The stdlib interaction is asserted from documentation
  today, not measured — if it degrades pooling, the field's meaning changes and the name is already fixed.

Non-goals for this slice

FastCGI/uWSGI (141-C); L4 (141-D); cross-protocol soak and BenchmarkAdmit_* (141-E).

Completion evidence

- PR:
- MaxConnsPerHost integration test:
- Per-protocol accounting matrix tests:
- Docs (accounting matrix + protocol applicability):

[juliaggas]

Implementation start

• Starting main SHA: 00e85c5a5ba06792b6aa5c43338d192f58a54294 (the merge of [RES-01-A] Resilience policy, admission primitive, and HTTP proxy integration #283)
• Governing ADR: ADR 0017 ([ADR][CGC-03] Specify generic resilience and overload-control architecture #116)
• Parent umbrella: [RES-01] Upstream admission: concurrency, pending, and connection limits #141, under epic [EPIC][RESILIENCE] Add bounded, protocol-consistent upstream resilience #110
• Hard predecessors: [RES-01-A] Resilience policy, admission primitive, and HTTP proxy integration #283 only, closed with evidence. This slice consumes its Admission primitive and resolved policy rather than adding a second mechanism.
• Branch: agent/issue-284-connection-bounds
• PR: feat(upstream): connection bounds and protocol lifetime accounting (#284) #300

Source re-audit

Verified against 00e85c5a rather than assumed:

• newProxyTransport(loc, policy) exists and sets MaxIdleConnsPerHost: 32 but no MaxConnsPerHost, exactly as the issue describes.
• NewGRPCProxy returns a bare *httputil.ReverseProxy — no wrapper type, and therefore no io.Closer, so its http2.Transport's idle connections were never dropped on generation retirement. The issue says native gRPC "has its own handler entry point and needs the Admit call"; what it does not say is that there was no seam to put it in, so one had to be created.
• Transcoder already implements http.Handler and io.Closer and is staged by internal/app/factory.go, so transcoding did have a seam.
• The health probe client is built by internal/upstream's own probeTransport, entirely separate from newProxyTransport. The exemption the issue requires is therefore structural already; what was missing was a test pinning it.

Scope

max_connections_per_backend with its pool and location scopes, admission for native gRPC and transcoding, the accounting matrix asserted per protocol, the lifecycle entries and the documentation. Nothing else — FastCGI/uWSGI (#285), L4 (#286) and the cross-protocol soak and BenchmarkAdmit_* (#287) stay in their own slices.

Differences from historical issue assumptions

1. The location block is a distinct Go type, LocationResilienceConfig, carrying only the stateless controls. [RES-01-A] Resilience policy, admission primitive, and HTTP proxy integration #283 deferred the location-level field because four schema leaves whose only behaviour is rejection would have forced the first entries into SchemaExemptions. Splitting the type solves it properly: the scope rule is enforced by strict decoding, the closed world gains exactly two leaves and both have real consumers, and the exemption map stays empty.

2. A location value of 0 inherits the pool's rather than meaning "unlimited". The ADR gives the field scope "transport, host" and says zero means unlimited, but says nothing about how a location zero composes with a pool value. Every other zero in this configuration means "not set", so inheriting is the only reading that does not surprise. The consequence — a route cannot be made explicitly unbounded under a bounded pool — is documented rather than left to be discovered.

3. Closing a retired generation now also drops the gRPC transport's idle HTTP/2 connections. Not in the issue, but the admission wrapper has to implement io.Closer for the retirement wakeup, and leaving the sockets open once the seam exists would be a gratuitous omission.

4. The WebSocket row's real semantics differ from the naive reading. Go 1.26's ReverseProxy.handleUpgradeResponse waits for the second copier when the first ends without an error, and io.Copy returns nil at EOF. A backend closing its half therefore does not end the tunnel, and the slot stays held until the connection really ends. The first version of the test asserted the naive behaviour and failed; the code was right and the expectation was wrong. Both halves are now pinned explicitly and the property is documented, because it is exactly the shape that gets reported as a leak.

Verification plan

Per-protocol accounting tests for HTTP/1.1 with and without keep-alive, concurrent HTTP/2 streams, real QUIC streams over one HTTP/3 connection, WebSocket for the full upgraded-connection lifetime, SSE, gRPC unary and streaming through a real h2 stack, and transcoding; client disconnect and panic release on each; a MaxConnsPerHost integration test that measures the bound and the keep-alive pooling interaction rather than trusting the standard library's documentation; the health-probe exemption; validation and lint tables; -race over every touched package including the tagged suites; and CI at the exact PR head.

[juliaggas]

Completion evidence

• Starting main SHA: 00e85c5a5ba06792b6aa5c43338d192f58a54294
• PR: feat(upstream): connection bounds and protocol lifetime accounting (#284) #300
• Final PR head: a188540b
• Merge SHA: 97e127c85bc8badc40e3b8192a6f71e5895a2d33

Primary packages/files

Area	Files
Connection bound	internal/handler/proxy.go (newProxyTransport), internal/resilience/resilience.go
Shared admission wrapper	internal/handler/admission.go (new)
Native gRPC	internal/handler/grpcproxy.go
Transcoding	internal/handler/grpctranscode.go
Configuration	internal/config/schema.go, validate_resilience.go, lint_resilience.go
Lifecycle	internal/lifecycle/registry.go

Public configuration changes

max_connections_per_backend, in two places: [upstreams.resilience] and [servers.locations.resilience], with the location winning. Default 0 is unlimited, so nothing changes for an existing configuration.

The location block is a new, distinct Go type (LocationResilienceConfig) carrying only the stateless controls. A stateful key written under a location is therefore rejected by strict decoding rather than by a validation rule, which is what #283 deferred and what keeps SchemaExemptions empty.

New lint warning: the bound set on a route that reaches its backend over one multiplexed HTTP/2 connection — native gRPC or transcoding — where it bounds nothing.

API changes

None.

Compatibility impact

None for existing configurations. Two behaviour changes, both improvements and neither reachable without new configuration:

• Native gRPC and transcoding routes now consume pool admission. With no max_active_requests set (the default) nothing is ever rejected, so this is observable only as a counter.
• Retiring a generation now closes the gRPC transport's idle HTTP/2 connections, which nothing previously did.

newProxyTransport gained a parameter; it is unexported.

Lifecycle changes

Two entries, both hot_reload under the resilience subsystem, with a reason specific to transport-scoped values rather than reusing the policy-swap reason: the bound is a property of the outbound transport, which is already rebuilt with the generation that owns it.

Closed world preserved: 260 leaves, 260 entries, 0 exemptions.

Generated artifacts

docs/config-lifecycle.yaml, docs/generated/config-lifecycle.json, docs/generated/config-lifecycle.md regenerated; make generated-check clean on merged main. docs/config-value-contract.json gained two audited dispositions.

Focused tests

MaxConnsPerHost integration test — TestMaxConnsPerHostBoundsSockets counts real accepted connections against a real backend. It proves the bound holds under 12 concurrent requests with admission unlimited, and then proves 30 sequential keep-alive requests open zero new sockets, which is the MaxIdleConnsPerHost: 32 interaction the issue asked to be measured rather than taken from documentation. TestMaxConnsPerHostLocationOverridesPool covers the five scope combinations.

Per-protocol accounting matrix:

Row	Test
HTTP/1.1, keep-alive	TestAccountingHTTP11KeepAlive — 20 requests on one connection, quiesces at 0
HTTP/2, +1 per stream	TestAccountingConcurrentStreams
HTTP/3, +1 per stream	TestHTTP3AdmissionCountsPerStream (internal/server, http3 tag) — real QUIC: two concurrent streams on one connection take two slots, a third is rejected
Protocol independence	TestAccountingProtocolAgnostic — identical accounting for HTTP/1.1, HTTP/2.0, HTTP/3.0
WebSocket / 101	TestAccountingWebSocketHoldsSlotForConnectionLifetime
Server-sent events	TestAccountingSSEHoldsSlotForResponseLifetime
gRPC unary	TestGRPCAccountingUnary
gRPC streaming	TestGRPCAccountingStreamHoldsSlotForLifetime — bidirectional, plus a second concurrent stream on the same connection
gRPC transcoding	TestTranscodeAccountingReleasesOnEveryPath, TestTranscodeAccountingEnforcesLimit
Client disconnect	TestAccountingClientDisconnectReleasesSlot, TestGRPCAccountingCancelReleasesSlot
Panic	TestAccountingPanicReleasesSlot
Shared wrapper lifetime	TestAdmittedHandlerHoldsSlotForHandlerLifetime, ...ReleasesOnceOnCancel, ...CloseWakesQueued, ...CloseRunsWrappedCloser
Health-check exemption	TestHealthProbeTransportIsExemptFromConnectionBound (internal/upstream)
gRPC overload status	TestGRPCAccountingOverLimitIsUnavailable — UNAVAILABLE, not RESOURCE_EXHAUSTED

Configuration: TestValidateMaxConnectionsPerBackend, TestParseRejectsStatefulControlsInLocation (all four stateful keys), TestLintWarnsConnectionBoundOnMultiplexedRoute (pool-level, location-level, and a silent HTTP/1.1 case).

Race evidence

go test -race over internal/handler internal/upstream internal/config and over internal/handler internal/server with -tags "grpc http3" — clean locally. CI test -race pass (3m43s) at PR head a188540b.

Property/fuzz evidence

No new targets. CI fuzz (smoke) pass (9m6s).

Protocol/E2E

Native gRPC unary and bidirectional streaming through a real HTTP/2 stack; transcoding through the real handler; real QUIC for HTTP/3; a real hijacked 101 upgrade and a real SSE stream. console e2e (browser smoke), console e2e (real server), consul live convergence and k8s kind live convergence all pass.

Benchmark evidence

benchmarks (smoke) pass. No numeric gate is claimed; that is #287's, and it must first introduce the proxy round-trip benchmark PRE-0 showed does not exist.

Soak evidence

soak (smoke) pass. The long-running resilience soak belongs to #287 and #144.

CI/checks

All 35 checks pass at PR head a188540b, including DCO, gofmt, golangci-lint, govulncheck, license-check, docs-check, full-tags-sync, coverage gate, security package coverage, security negative tests (lean/full), test (lean/full), test -race, test-macos (lean/full), test-windows (lean/full), build (lean/full) and CodeQL for Go, JS/TS, Python and actions. Merged main re-verified locally with make ci-fast and make generated-check.

Documentation

docs/upstreams.md gains the normative accounting matrix, a "requests versus sockets" section, the max_connections_per_backend reference with both exemptions stated, and the location-scope rule. docs/configuration.md gains the key. scripts/docs-check.py: 1298 passed, 0 failed.

Changelog

One Added entry under [Unreleased].

Security review

No new attack surface. The bound is an upper limit on sockets Jul opens, so it can only reduce outbound resource use. Admission on gRPC and transcoding rejects with a fixed status and no operator- or peer-controlled string in any header, log or metric. The health-check exemption is the one place where a limit is deliberately not applied, and it is a safety property rather than a hole: without it a saturated pool could never observe a backend recovering, so a transient overload would become permanent.

Cardinality review

No metrics in this slice.

Deviations from issue assumptions

1. NewGRPCProxy had no seam at all. The issue says native gRPC "has its own handler entry point and needs the Admit call". In fact it returned a bare *httputil.ReverseProxy — not a named type, and not an io.Closer — so a wrapper had to be created rather than a call added. The wrapper is shared with transcoding, which did already have a seam.

2. The location block is a separate type. LocationResilienceConfig carries only the stateless controls. The alternative — one shared type with a validation rule rejecting the stateful keys — would have added four schema leaves whose only behaviour is rejection, forcing the first entries into SchemaExemptions. Encoding the scope rule in the type means strict decoding enforces it and the exemption map stays empty.

3. A location value of 0 inherits the pool's, rather than meaning "unlimited". The ADR does not say how a location zero composes with a pool value; every other zero in this configuration means "not set", so inheriting is the only reading that does not surprise. Documented, with its consequence: a route cannot be made explicitly unbounded under a bounded pool.

4. Closing a retired generation now drops the gRPC transport's idle HTTP/2 connections. Not requested. The wrapper must implement io.Closer for the retirement wakeup, and leaving the sockets open once the seam exists would have been a gratuitous omission.

5. The WebSocket row's real semantics are not the naive reading, and the first test asserted the naive one and failed. Go 1.26's ReverseProxy.handleUpgradeResponse ends with err := <-errc; if err == nil { err = <-errc }, and io.Copy returns nil at EOF — so a clean half-close waits for the second copier. A backend closing its side while the client's half is still open has not ended the tunnel, and the slot is correctly still held. The code was right; the expectation was wrong. The test now pins both halves and docs/upstreams.md calls it out, because it is exactly the shape that gets reported as a leak.

Residual limitations

• FastCGI and uWSGI do not participate in admission yet ([RES-01-C] FastCGI and uWSGI pool membership, Backend.Network, and weight-stable identity #285), nor does L4 ([RES-01-D] L4 TCP admission, stream-owned upstream registry, and tcp-forced stream health #286).
• The bound is per transport, and a transport is per location. Two locations pointing at the same upstream therefore get one bound each rather than one shared bound. This follows from MaxConnsPerHost being the only lever Go offers and is what the field name says; it is not a pool-wide socket cap and cannot be.
• Idle connections count toward the bound until IdleConnTimeout (90s, unchanged), so a route that has just finished a burst can briefly refuse to open new sockets.
• No metric for connections yet; jul_upstream_connections belongs to [RES-04] Resilience failure taxonomy, bounded metrics, API/Console surfaces, and soak closure #144.

Acceptance criteria

• MaxConnsPerHost wired from the resolved policy; warns on gRPC and transcoding routes; health-check clients exempt.
• Native gRPC and transcoding acquire and release admission on every terminal path, including cancelled streams. Half-open is covered by the WebSocket half-close case and by the gRPC cancellation case.
• Every row of the accounting matrix is asserted by a test.
• A long-lived stream holds its slot for its full lifetime and releases exactly once.
• Lifecycle registry entries for max_connections_per_backend, docs for the accounting matrix.

Next issue

#285 (RES-01-C) — FastCGI/uWSGI and the backend model.