Restore cluster RPC fabric using WebSocket transport (working name: "Elemental Conduit")

[ptwohig]

Background: the cluster RPC fabric (the rt-* modules, built on JeroMQ) was intentionally disabled for the 3.0 release -- not because it was broken, but because there wasn't a strong enough business case to keep maintaining it at the time. It's needed again for upcoming requirements. Four planning docs (CLUSTER-RPC-PLAN.md, RT-TRANSPORT-COMPARISON.MD, CLUSTER-CAPACITY.md, APP_NODE_IMPLEMENTATION_PLAN.md) worked through renovating the existing JeroMQ transport vs. replacing it with a Jetty WebSocket transport. This issue tracks executing the WebSocket path, which is what those docs converge on.

Why WebSocket over renovating JeroMQ: in how this codebase actually uses ZMQ (DEALER/ROUTER as a glorified TCP socket with length-prefixed framing and routing-identity envelopes -- not PUB/SUB, not multi-peer fanout, not inproc://-as-IPC), WebSocket is a strict superset plus a free HTTP layer: auth via the existing session-secret/OIDC pipeline, TLS via the existing Jetty SSL config, standard close codes, L7 proxy/mesh compatibility, ping/pong liveness, and observability that already works for the REST API. JeroMQ's threading model fights Java's concurrency primitives directly (documented as the "threading nightmare"), is pinned to an old version because upstream 0.6.0 hangs under our usage pattern, and the effort/risk estimates for renovating it in place are worse on every axis (RT-TRANSPORT-COMPARISON.MD §4-6).

Plan (phases, condensed from APP_NODE_IMPLEMENTATION_PLAN.md §5 -- see that doc for full detail and the resolved open-questions log in §6):

0. Shared safety net. Timeouts + leak assertions on the existing (currently @Test(enabled = false)) JeroMQ integration tests. Sunk cost regardless of anything else in this plan.
1. WS transport prototype. cluster-ws/cluster-ws-guice stubs; one static wss://host/cluster/v1 endpoint, registered once at bootstrap in deployment-jetty. Round-trip a single sync invocation against the existing (deprecated) dispatchers first, to validate the transport in isolation before touching dispatch. Decision point: if this round-trip doesn't land cleanly, fall back to the JeroMQ renovation path in JEROMQ.md instead of continuing.
2. Invocation semantics parity. Multiplexing by request id, async / CompletionStage invocations, best-effort cancellation (interrupt server-side, at-most-once delivery client-side), per-call timeouts, a per-peer session cache with lazy dial + coalesced concurrent dials, a configurable graceful-drain timeout (default 30s), and a 10 MiB per-frame cap. No outbox, no reconnect-with-buffering -- K8s SRV-gated discovery plus dial-on-demand plus registry health-pruning already cover what buffering would have bought us, and buffering would actively conflict with the health-pruning contract.
3. ElementRegistry-based dispatch. Replace the three deprecated LocalInvocationDispatcher implementations with one that resolves through ElementRegistry, via a new DeploymentId (deterministic hash of a deployment's own id) as an O(1) key into activeDeployments, with a merge-scan fallback across ElementRuntimeService.getActiveRuntimes() when wildcarded. ApplicationId-based scoping is dropped as legacy.
4. ClusterElementLocator + addressing. New ElementAddress type in sdk-cluster, built from sdk-cluster's existing Path type via composition (not subclassing -- Path is final), reusing ElementServiceKey's #-separated encoding for service+name. Explicit per-method @RemotelyInvokable opt-in (default-deny) -- @ElementService alone does not make a method remotely callable.
5. Auth. No new auth scheme needed. The WS upgrade handshake carries the existing SessionSecretHeader, validated through the existing SessionDao/SessionService chain against the shared MongoDB (every instance already has DB access, so the database is already the shared trust anchor). User-initiated cross-instance calls forward the caller's real session; pure control-plane calls (health/discovery) use a provisioned per-instance system/service User + minted Session.
6. Discovery. Plug the existing SRV-backed InstanceDiscoveryService implementations into the new WS registry.
7. Coexistence + cutover. elements.rt.transport config flag, default jeromq for one release then ws, both exercised in CI during the transition.
8. Retire JeroMQ. Delete the six rt-*-jeromq modules and related config; keep JEROMQ.md for historical reference.
9. Observability + tuning. Per-peer connection metrics, idle-timeout / ping-interval tuning, published /cluster/v1 frame-format docs.
10. Retire app-node. Fold cluster-node hosting into jetty-ws behind a new operational-mode CLI flag; delete the separate app-node executable (already confirmed dead/broken code, unmaintained for a long time).

Independent side-quest, land regardless of transport outcome: replace the unimplemented InstanceMetadataContext.getInstanceQuality(): double score (no production implementation exists today -- the only impl in the tree is a test mock returning a random double) with concrete available/total resource accounting: a ResourceBudget/ResourceClaim SPI, local optimistic debit between the registry's ~30s refresh cycles, sorting candidates by absolute available headroom (not ratio), and a configured (not auto-detected) total per instance. Full design in CLUSTER-CAPACITY.md. This lives in rt-server-cluster, is transport-independent, and should land early since it removes a known-broken piece from production sooner.

Naming. New modules drop the rt-server-/rt- prefix (cluster-ws, cluster-ws-guice, cluster-node-ws); the URL path becomes /cluster/v1/... (not /rt/v1/...). rt-*-jeromq modules keep their names through retirement to avoid rename churn on code that's going away anyway.

Suggested feature name: "Elemental Conduit" -- a conduit carries something between two points, which is exactly what this is (cluster RPC transport), and it pairs naturally with the existing "Elements" branding. Alternates considered: "Elemental Current", "Elemental Circuit".

References: CLUSTER-RPC-PLAN.md, RT-TRANSPORT-COMPARISON.MD, CLUSTER-CAPACITY.md, APP_NODE_IMPLEMENTATION_PLAN.md (all at repo root).

[ptwohig]

Issue #10 — Second Stage: Real Cluster Transport, Dual-Mode Launch, RemoteElementRegistry

Set aside on 2026-08-17 to work on 3.9.0-rc-1 bug fixes first. Resume here when ready — this content is unchanged from where it was left.

Status of prior work

Phase 0 (JeroMQ removal) and Phase 1 (Fabric WebSocket transport prototype) are done and merged: elements PR #31, elements-manual PR #2. That work proved the WS transport mechanics in isolation (cluster-fabric/cluster-fabric-guice, FabricEndpoint/FabricRemoteInvoker, registered once at bootstrap via deployment-jetty's FabricWsBootstrap) but left it disconnected from everything else: a point-to-point invoke(URI, Invocation) call with no NodeId awareness at all.

Context

Investigating "does the system still route by NodeId" surfaced that the surviving cluster machinery (RemoteInvocationDispatcher → RoutingStrategy → RemoteInvokerRegistry → InstanceConnectionService → RemoteInvoker) is real, production-wired Guice-graph code, not dead scaffolding — but two of its links (InstanceConnectionService, RemoteInvoker) have zero implementations left after JeroMQ's deletion, and the SDK has no RemoteElementRegistry-style façade over any of it (ElementRegistry is purely local, no remote concept whatsoever). Separately, app-node/ApplicationNodeMain — the old standalone worker-node binary — is confirmed dead weight (everything in it is either duplicated library-module composition or already broken post-JeroMQ), and Elements has no way today to actually run as a cluster worker versus an external-facing web service from the same jetty-ws binary.

This stage closes all of that in one PR (explicit direction: don't split it up, this is overdue technical debt): a semver-safe ApplicationId→DeploymentId rename, deletion of app-node, a real Jakarta-WebSocket-backed InstanceConnectionService/RemoteInvoker pair replacing JeroMQ's, a new SDK-level RemoteElementRegistry, and a --mode external|worker dual-launch flag on ElementsMain.

Explicitly deferred to later work (confirmed with user):

• ClassLoader isolation for the worker's TransactionalResourceServicePersistence — for now it's a plain Guice-bound service sharing the worker's root classpath (matching how app-node's UnixFSStorageDriverModule already composed it), not a real isolated Element. No mechanism exists today to load a bundled/built-in Element without the git/ELM artifact pipeline; building that is separate future work.
• Full per-NodeId addressing/ClusterElementLocator (original roadmap's Phase 4) — RemoteElementRegistry in this stage resolves by DeploymentId only (one connection per discovered instance, via RemoteInvokerRegistry.getBestRemoteInvoker(DeploymentId)), not by specific NodeId.

---

Phase A — Rename ApplicationId → DeploymentId (semver-safe, do first)

sdk-cluster's ID types are part of the versioned SDK contract Elements compile against, so this can't be an in-place rename. Follow the exact precedent already in this codebase (rt-common/.../util/Hex.java extends dev.getelements.elements.sdk.util.Hex {} — empty @Deprecated subclass pointing at the new home), adapted for ApplicationId's real constructors/static factories:

1. New sdk-cluster/src/main/java/dev/getelements/elements/sdk/cluster/id/DeploymentId.java — move every field/constructor/method from ApplicationId.java here verbatim, renamed where natural (getDeploymentUUID(), randomDeploymentId(), forUniqueName(String), valueOf(String)). Change equals() from ApplicationId's exact-class check (ApplicationId.class.equals(o.getClass())) to instanceof DeploymentId — otherwise a DeploymentId and an ApplicationId holding the same UUID won't be .equals() during the migration window, which would silently break map-key lookups anywhere both types end up mixed.
2. ApplicationId.java → @Deprecated public class ApplicationId extends DeploymentId, Javadoc @deprecated use {@link DeploymentId}. Re-declare its 4 constructors (delegating to matching super(...)) and 3 static factories (randomApplicationId(), forUniqueName(String), valueOf(String)) so they still return ApplicationId, not DeploymentId. Keep getApplicationUUID() as a deprecated delegate to getDeploymentUUID().
3. NodeId.java — add primary getDeploymentId(): DeploymentId (replaces the internal applicationId cache field's role) and forInstanceAndDeployment(InstanceId, DeploymentId). Keep getApplicationId(): ApplicationId and forInstanceAndApplication(...) as @Deprecated delegates (works since ApplicationId is-a DeploymentId).
4. Repoint all 19 internal production call sites directly to DeploymentId (mechanical — these are internal runtime-module signatures, not the frozen public type itself, matching "internally, re-point all constants and terminology"): ApplicationAssetLoader (sdk-cluster), Context/Context.Factory (rt-common), RemoteInvokerRegistry (rt-common), AbstractAggregateRoutingStrategy/DefaultRoutingStrategy (rt-common), CachingContextFactory/RemoteInvokerRegistrySnapshot/SimpleRemoteInvokerRegistry (rt-server-cluster), ClusterContextFactoryModule/NodeIdModule (rt-server-cluster-guice), Node/Node.Factory/SimpleWorkerInstance/Worker/Worker.Mutator (rt-server-cluster-node), GitApplicationAssetLoader/FilesystemGitApplicationAssetLoader (rt-server-git), EmbeddedWorkerInstanceContainer/TestClientContextFactoryModule (rt-server-testkit), FileSystemApplicationRepositoryResolver (common-git). Do not touch sdk-model's Application/ApplicationDao or the manual's security-model.md "ApplicationId" REST field mention — confirmed unrelated naming collisions with the domain entity, not this ID type.

Verification: mvn -pl sdk-cluster,rt-common,rt-server-cluster,rt-server-cluster-guice,rt-server-cluster-node,rt-server-git,common-git,rt-server-testkit -am install; existing ApplicationIdTest/NodeIdTest/ResourceIdTest/TaskIdTest/SimpleRemoteInvokerRegistryTest must pass unmodified, proving compatibility held.

---

Phase B — Delete app-node

Confirmed via full audit: everything in it is either already-broken post-JeroMQ (MasterNodeModule's circular Node binding, WorkerInstanceModule's commented-out git loader + Node.Factory.unsupported()) or pure composition of library modules with zero unique logic (AppNodeServicesModule ≈ existing ServicesElementModule; UnixFSStorageDriverModule is a two-line passthrough of JournalTransactionalResourceServicePersistenceModule + UnixFSTransactionalPersistenceContextModule, both reused directly in Phase F). HACKING.md already documents it as "deprecated and slated for removal." No Dockerfile/docker-compose references it; jetty-ws-test/pom.xml has a dependency on it with zero source-level usage.

1. git rm -r app-node.
2. Root pom.xml: remove <module>app-node</module> and its <dependencyManagement> entry.
3. jetty-ws-test/pom.xml: remove the unused app-node dependency.
4. HACKING.md: replace the "deprecated, slated for removal" note with a short line that it's gone, folded into jetty-ws's --mode flag (Phase F).

PersistentInstanceIdModule (rt-server-cluster-guice) and the transact/UnixFS modules survive as library code — Phase F reuses them directly, so nothing needed here is stranded.

---

Phase C — Final JeroMQ-vestige cleanup pass

Confirmed orphaned (zero remaining implementations anywhere): rt-common/src/main/java/dev/getelements/elements/rt/AsyncConnection.java, AsyncConnectionPool.java, AsyncConnectionGroup.java, AsyncConnectionService.java, SharedAsyncConnectionService.java — generic (non-JeroMQ-named) connection scaffolding that only JeroMQ's deleted classes ever implemented. Delete these five files. This phase is a sweep, not exhaustively pre-enumerated: also grep for other now-dead JeroMQ-shaped leftovers (stray config key constants, doc mentions) at implementation time and remove what's confirmed dead.

---

Phase D — Real transport: JakartaWebsocketRemoteInvoker + JakartaWebsocketInstanceConnectionService

This closes the actual "does NodeId routing work" gap. Both new classes live in cluster-fabric (add a dependency on rt-server-cluster for InstanceConnectionService/InstanceDiscoveryService/InstanceHostInfo).

Explicit design correction (per direction): no persistent-connection layer at all. JeroMQ's two-tier connection model (a long-lived control connection per instance, plus a separately-negotiated per-node route/RemoteInvoker pair) existed specifically to work around ZMQ's connection-setup cost and DEALER/ROUTER semantics. Jakarta WebSocket has no equivalent cost that needs working around — each invocation dials its own fresh WebSocket connection, exchanges exactly one request/response, and closes, trusting the underlying WS client for whatever pooling/reuse it already does under the hood. No manual connection caching, no multiplexing, no request-id correlation needed (a connection carries exactly one call, so whatever comes back on it is unambiguously the answer).

1. Rename FabricRemoteInvoker → JakartaWebsocketRemoteInvoker, make it a real RemoteInvoker.

Per direction: @ClientEndpoint-annotated (declarative @OnOpen/@OnMessage/@OnError methods, not the current hand-built Endpoint subclass + ClientEndpointConfig), depending only on jakarta.websocket interfaces. Implements RemoteInvoker's genuinely-abstract surface: getConnectAddress(), start(String connectAddress, long timeout, TimeUnit), invokeAsync(Invocation, List<Consumer<InvocationResult>>, InvocationErrorConsumer): AsyncOperation, invokeCompletionStage(Invocation, ...): CompletionStage<Object> — everything else (invokeSync, invokeFuture, invokeAsyncV, the deprecated invoke(...)) is already a default method layered on invokeCompletionStage. Read AsyncOperation's actual interface before implementing invokeAsync — not yet read in this research pass, needed for that one method.

• start(connectAddress, timeout, unit) is trivial: just records the address on the instance. No network I/O happens here, no readiness blocking — it's eliminated as a real lifecycle step, kept only because it's an existing abstract method on the shared RemoteInvoker interface that SimpleRemoteInvokerRegistry still calls once per NodeId.
• Each invokeCompletionStage(...) call is fully self-contained: dial a fresh WS connection to the stored address (ContainerProvider.getWebSocketContainer().connectToServer(...)), send the one Invocation frame from @OnOpen, receive the one response frame via @OnMessage, close the session, complete the returned CompletionStage. Since RemoteInvokerRegistry/routing strategies may call the same cached JakartaWebsocketRemoteInvoker instance concurrently for the same NodeId, the per-call dial must be isolated per invocation (e.g. a small internal holder object constructed fresh for each call, holding its own Session/CompletableFuture state) rather than instance fields shared across calls — the outer JakartaWebsocketRemoteInvoker object itself stays effectively stateless past the stored address.
• FabricInvocationFrame needs no changes — no request-id needed, since one connection = one call.

2. New JakartaWebsocketInstanceConnectionService implements InstanceConnectionService.

The interface itself is unchanged from before JeroMQ (only its implementation was deleted): start()/stop()/refresh()/getInstanceId()/openBinding(NodeId): InstanceBinding/getActiveConnections(): List<InstanceConnection>/subscribeToConnect/subscribeToDisconnect/getLocalControlAddress(), with a nested InstanceConnection (getInstanceId()/getInstanceMetadataContext()/openRouteToNode(NodeId): String/disconnect()).

Given invocations no longer need a persistent connection at all, this service's job shrinks to address-book/metadata tracking, not connection lifecycle management: for each InstanceHostInfo InstanceDiscoveryService reports (getKnownHosts() + subscribeToDiscovery/subscribeToUndiscovery), it tracks that instance as "known," periodically refreshing its metadata (InstanceId, hosted NodeIds, quality) via a one-shot call — dial, ask, close, exactly like a data invocation — rather than holding anything open between refreshes.

• Interface change (internal, rt-server-cluster, one real caller — safe): replace InstanceConnection.openRouteToNode(NodeId): String with InstanceConnection.getRemoteInvoker(): RemoteInvoker — returns a JakartaWebsocketRemoteInvoker already start()-ed against that instance's address (cheap, since start() is just recording the address). No per-node address minting.
• SimpleRemoteInvokerRegistry.establishNewConnection(NodeId, InstanceConnection) simplifies to return connection.getRemoteInvoker(); — drop the now-unused remoteInvokerProvider/start(addr) call and its Guice setter/field entirely. This is the one place in the whole existing RemoteInvokerRegistrySnapshot/SimpleRemoteInvokerRegistry machinery that changes; everything else (refresh scheduling, InstanceMetadata-driven NodeId discovery, per-NodeId snapshot storage) stays as-is.
• Metadata refresh: build an InstanceMetadataContext proxy over a one-shot JakartaWebsocketRemoteInvoker (new ProxyBuilder<>(InstanceMetadataContext.class).withHandlersForRemoteInvoker(invoker).build() — exactly what the deleted JeroMQInstanceConnection did, just without keeping the invoker's connection open afterward) to learn the remote's InstanceId and hosted NodeIds on each refresh cycle, then publish via the connect-subscription callbacks.
• openBinding(NodeId)/InstanceBinding: no per-node binding concept exists in the new model (one static /cluster/v1 endpoint, not per-node). Minimal implementation wrapping that one static path; revisit if per-node addressing (deferred, see Context) ever lands.
• Discovery address format: SpotifySrvInstanceDiscoveryService/JndiSrvInstanceDiscoveryService's getConnectAddress() currently emit JeroMQ-flavored tcp://host:port strings — update to ws://host:port/cluster/v1 (or wss://, matching JakartaWebsocketLoader's existing scheme-inference pattern) since that's what JakartaWebsocketRemoteInvoker now dials per call.
• Guice wiring: extend cluster-fabric-guice's FabricGuiceModule (or a sibling module) to bind InstanceConnectionService → JakartaWebsocketInstanceConnectionService and a Provider<RemoteInvoker> (consumed by the connection service to mint per-instance invokers).

---

Phase E — RemoteElementRegistry (SDK-level façade)

New interface, sdk/src/main/java/dev/getelements/elements/sdk/RemoteElementRegistry.java, the remote analog to ElementRegistry — the piece that was missing entirely. Proposed shape (flag for review, this is genuinely new API):

public interface RemoteElementRegistry {
    <T> Optional<T> findRemote(DeploymentId deploymentId, ElementServiceKey<T> key);
}

Implementation resolves RemoteInvokerRegistry.getBestRemoteInvoker(deploymentId) (works once Phase D lands), then builds/caches a dynamic proxy for the requested service via the existing ProxyBuilder/RemoteInvocationHandlerBuilder machinery — the same pattern RemoteProxyProvider already uses for the fixed set of @Named(REMOTE) contexts, generalized to an arbitrary interface + DeploymentId. Lives in rt-server-cluster-guice (needs RemoteInvokerRegistry), bound similarly to RootElementRegistryModule.

---

Phase F — Dual-mode jetty-ws launch (--mode external|worker)

ElementsMain already uses joptsimple (--web-service, repeatable ElementsWebServiceComponent enum, defaultsTo(values())) — add a sibling --mode option, new enum (EXTERNAL/WORKER), defaultsTo(EXTERNAL).

External mode (default): today's behavior, unchanged, plus it becomes a real cluster client instead of cluster-blind: RandomInstanceIdModule stays (fresh identity per process, matches "assigns itself a random InstanceId"). ElementsCoreModule keeps installing the transport-agnostic cluster block (InstanceDiscoveryServiceModule, ClusterContextFactoryModule, SimpleRemoteInvokerRegistryModule, KryoPayloadReaderWriterModule) plus the Phase D JakartaWebsocketInstanceConnectionService binding — external can now dial out and resolve remote proxies via RemoteElementRegistry. External does not install FabricWsBootstrap (no /cluster/v1 server) — split deployment-jetty's FabricWsModule so ElementsServletContextModule only installs the endpoint-registration half in worker mode; external gets just the transport bindings (FabricGuiceModule) for its client role.

Worker mode (new): exposes only the cluster WebSocket. A leaner Guice graph, not today's full ElementsCoreModule+JettySdkElementModule+ElementsWebServiceComponentModule stack:

• JettyServerModule (still needed to host the one WS endpoint).
• Cluster core: PersistentInstanceIdModule (not Random — "assigns an InstanceId from PersistentInstanceIdProvider", stable across restarts, exact precedent already in the deleted ApplicationNode), InstanceDiscoveryServiceModule, ClusterContextFactoryModule, SimpleRemoteInvokerRegistryModule, KryoPayloadReaderWriterModule, the Phase D JakartaWebsocketInstanceConnectionService binding.
• Fabric server: FabricWsModule's endpoint-registration half (FabricWsBootstrap), via a slimmed ElementsServletContextModule-equivalent Handler.Sequence assembly (no JakartaRsLoader/StaticContentLoader REST/static wiring — worker serves nothing but /cluster/v1).
• TransactionalResourceServicePersistence exposure: per direction, plain Guice, shared classpath (deferred ClassLoader isolation) — install JournalTransactionalResourceServicePersistenceModule + UnixFSTransactionalPersistenceContextModule + TransactionalResourceServiceModule directly, the exact same composition app-node's (now-deleted) UnixFSStorageDriverModule used, so ServiceLocatorLocalInvocationDispatcher/Fabric dispatch can resolve it when invoked remotely.

Both modes' cluster-core module list is identical except RandomInstanceIdModule vs PersistentInstanceIdModule and whether FabricWsBootstrap is installed — worth factoring into one shared ClusterCoreModule parameterized by instance-id strategy, rather than duplicating the list.

---

Verification

• Full tree: mvn clean install -DskipTests -Dmaven.javadoc.skip=true.
• ApplicationIdTest/NodeIdTest/ResourceIdTest/TaskIdTest/SimpleRemoteInvokerRegistryTest pass unmodified (Phase A compatibility).
• New decision-gate integration test (mirrors Phase 1's original round-trip test, now end-to-end): launch one jetty-ws process/injector with --mode worker (static InstanceDiscoveryService pointing nowhere, just proving the worker graph boots and /cluster/v1 is live and TransactionalResourceServicePersistence is bound) and one with --mode external whose static discovery points at the worker; from the external side, obtain a proxy via RemoteElementRegistry.findRemote(...) and exercise a real @RemotelyInvokable call end-to-end through InstanceDiscoveryService → JakartaWebsocketInstanceConnectionService → SimpleRemoteInvokerRegistry → the proxy → a real result.
• mvn -pl service-test test still green, no regression.

[ptwohig]

App Node Implementation Plan: WS Transport + Element-Based Dispatch

Companion to JEROMQ.md and WEBSOCKETS.MD. Those two documents cover the transport decision (JeroMQ renovation vs. WebSocket replacement) in detail; this document is the synthesis: it combines the transport plan with two additional pieces surfaced while validating it against the actual codebase — the ElementRegistry dispatch gap and a proposed ClusterElementLocator addressing scheme — into one implementation plan for restoring the cluster RPC fabric (app-node).

Do not duplicate JEROMQ.md/WEBSOCKETS.MD's pros/cons analysis here; this document assumes the WS-transport direction from WEBSOCKETS.MD and focuses on what else has to be true for app-node to actually work end to end.

---

1. Why this document exists

Reviewing WEBSOCKETS.MD against the live code surfaced two things it doesn't account for:

1. The invocation pipeline it calls "already validated" is mid-migration. All three concrete LocalInvocationDispatcher implementations in production use are marked @Deprecated with an explicit repo-authored note that they must be replaced by Element-dispatch via ElementRegistry. WEBSOCKETS.MD's claim that the pipeline above the transport is stable is true of the current (deprecated) code, but the transport work and this pending migration land in the same call path and need to be sequenced deliberately, not discovered by accident mid-implementation.
2. ElementRegistry-based dispatch needs an addressing scheme, since remote calls need to name an Element, a service within it, and optionally a specific deployment/instance — none of which existed as a resolvable client-side construct before this plan. A ClusterElementLocator interface (living in sdk-cluster, not sdk — see §4) plus reuse of the existing Path type is proposed below, checked against existing prior art (Path, ElementServiceKey, ElementQuery, ProxyBuilder).

Both are new surface area beyond what WEBSOCKETS.MD scoped. This document folds them into one plan.

---

2. End-to-end invocation flow (target state)

Caller code
  → ClusterElementLocator.getRemote(ElementAddress, Class<T>)          [NEW — ElementAddress wraps a validated Path]
  → dynamic proxy (Proxy.newProxyInstance, cf. ProxyBuilder)  [existing pattern, rt-server-cluster]
  → RemoteInvoker.invokeAsync / invokeCompletionStage         [existing SDK contract, rt-common]
  → WsRemoteInvokerRegistry                                    [NEW — WS analog of SimpleRemoteInvokerRegistry]
      resolves target InstanceId (from the Path's authority, or "*" for any) against a live snapshot,
      built from InstanceDiscoveryService + health-pruning on failed peers
  → WsRemoteInvoker                                            [NEW]
      dials (or reuses) a Session to the peer, keyed by peer InstanceId;
      serializes Invocation via PayloadWriter into a WsInvocationFrame
  ⇢ wss://host/cluster/v1  (single, static, fixed URL — identical for every instance; no per-node/per-deployment
                             path segment. Which instance you're talking to is determined entirely by which
                             connection you dialed, not by anything in the URL. See §6 Q1, revised.)
  → WsRtEndpoint                                               [NEW]
      decodes frame via PayloadReader
      resolves SessionSecretHeader (from the WS upgrade request) via existing SessionService/SessionDao → access-level scope  [see §6 Q2]
      resolves Path's DeploymentId segment → O(1) lookup in activeDeployments (or merge-scan across
      getActiveRuntimes() if wildcarded) → ElementQuery(registry, elementName, ordinal) → Element → Service  [NEW — see §3, §6 Q8]
  → LocalInvocationDispatcher.dispatch(...)                    [existing, but backing impl changes — see §3]
  → target Element's exported service method executes on Jetty's QueuedThreadPool
  ⇠ result/error serialized via PayloadWriter, tagged with request id, written back on the same Session
  → WsRemoteInvoker resolves the pending CompletableFuture/consumer by request id
  → back to caller

Everything above the "WS transport" line (RemoteInvoker, LocalInvocationDispatcher's existence, SimpleRemoteInvokerRegistry's pruning behavior) is existing, verified code. Everything marked [NEW] is what this plan has to build. The ElementRegistry resolution step inside WsRtEndpoint has no JeroMQ-era equivalent to port forward — JeroMQNode never built it either, because the deprecated dispatchers didn't need it.

---

3. The ElementRegistry dispatch gap

3.1 Current state (verified against code)

• AbstractLocalInvocationDispatcher.dispatch() (rt-server-cluster-node) does Class.forName(invocation.getType()) then delegates to an abstract resolve(...) hook.

• Three concrete implementations exist, each wired per-context via Guice, and each carries the identical Javadoc:

  /**
   * @deprecated This needs to be replaced with a dispatch to an {@link Element} via {@link ElementRegistry}
   */
  @Deprecated

  • ServiceLocatorLocalInvocationDispatcher → ServiceLocator.getInstance(type[, name])
  • ContextLocalInvocationDispatcher → hard-coded switch over Context types (IndexContext, ResourceContext, etc.)
  • MasterNodeLocalInvocationDispatcher → resolves InstanceMetadataContext only

• JeroMQNode injects LocalInvocationDispatcher directly and calls dispatch() with no registry lookup of its own (JeroMQNode.java:394-406). There is no "hidden" ElementRegistry step to replicate from the JeroMQ code — it was never built there.

• ElementRegistry / MutableElementRegistry (sdk) and their root/scoped implementations (sdk-spi: RootElementRegistry, ElementScopedElementRegistry) already exist and are used today by the HTTP/deployment stack (deployment-jetty, jetty-ws, common-servlet) — just not by the RPC path.

3.2 What has to be built

• A new LocalInvocationDispatcher implementation that resolves through ElementRegistry instead of ServiceLocator/hard-coded Context switches.
• Resolution scope (see §6 Q8 for full detail): not ApplicationId-based — that's a legacy concept superseded by Path-based element addressing. Each deployment gets its own subordinate ElementRegistry (StandardElementRuntimeService.doLoadDeployment → RootElementRegistry.newSubordinateRegistry()), keyed by the deployment's own id in activeDeployments. Confirmed: a parent registry's stream() does not aggregate elements registered into its subordinates, so there's no existing flattened view to query. Resolution needs a new DeploymentId (a hash of the deployment id, mirroring ApplicationId.forUniqueName(...)'s existing derivation pattern) as an O(1) key into activeDeployments, with a merge-scan across ElementRuntimeService.getActiveRuntimes() as the fallback when the Path's deployment segment is wildcarded.
• WsRtEndpoint (or whatever replaces it) needs to do this resolution before calling dispatch(...), using the Path addressing scheme in §4 to find the right deployment, Element, and exported service.
• Known limitation: transient deployments (no stable, persisted id) are not addressable through this scheme — DeploymentId assumes a durable identifier. This is a deliberate scope boundary of the addressing design, not an oversight to fix later.

3.3 Sequencing risk

This migration and the WS transport work touch the same call path (Node/WsRtEndpoint → LocalInvocationDispatcher). Two viable orders:

1. Dispatch first: land ElementRegistry-based dispatch behind the existing JeroMQ transport first, prove it with the current (soon-to-be-retired) transport, then swap transports underneath a working dispatch layer.
2. Transport first: build the WS round-trip (WEBSOCKETS.MD Phase 1) against the existing deprecated dispatchers to validate the transport in isolation, then swap the dispatcher implementation once the transport is proven.

Recommendation: Transport first. WEBSOCKETS.MD's Phase 1 decision point (does the WS round-trip work with existing serializers?) is cheaper to evaluate against a known-quantity dispatcher. Bolting two unproven things together (new transport + new dispatch resolution) makes a Phase-1 failure ambiguous — you won't know which layer broke.

---

4. ClusterElementLocator and Path-based addressing

4.1 Proposal

SIGNED OFF (revised, resolves §4.3 item 1). Two-type design, settled after checking Path's actual source:

1. sdk-cluster's Path is reused completely unmodified as the wire/serialization representation. No subclass, no new methods on it, no domain-overload concern — it stays exactly what it is today (Resource/Handler/transaction-log addressing already uses it; this just adds one more thing that happens to fit its generic context://component/component/... shape). Note: Path is declared public final class Path — subclassing it was never legal Java to begin with, which settles that sub-option on its own.
2. A new type, ElementAddress (name tentative), constructed from a Path via a validating factory: ElementAddress.from(Path) (defaults to Thread.currentThread().getContextClassLoader()) or ElementAddress.from(Path, ClassLoader) for explicit control. It parses and checks the path's components once at construction — right segment count, well-formed deployment/element/service parts — and throws immediately if malformed, rather than letting a bad address surface as a runtime failure deep in dispatch. Its fields are fully parsed, typed values, not raw strings: notably getServiceKey(): ElementServiceKey, parsed via a new ElementServiceKey.tryParse(ClassLoader, String) overload (added alongside the existing ElementRecord-based one, which can't be used client-side — see §6.1.11) rather than a second hand-rolled #-split. getServiceKey().type() already exposes the resolved Class<?>, so no separate field is needed for that. ClusterElementLocator.getRemote(ElementAddress, Class<T>) and the server-side resolution in WsRtEndpoint both consume this pre-checked handle, not a raw Path.

Module placement (resolves §4.3 item 1's module question): ClusterElementLocator and ElementAddress both live in sdk-cluster, alongside Path/InstanceId. sdk-cluster already depends on sdk, so this is a normal downstream dependency, not a new one — and it settles the circular-dependency problem for good, since plain sdk never needs to reference sdk-cluster types. Plugins that want to make remote-element calls add sdk-cluster as a dependency alongside sdk.

Mapping onto Path's actual shape: Path's context field is the instance authority; its components list carries [deploymentId, elementName[ordinal], serviceKey] in order — no new grammar needs inventing, it's Path used exactly as designed:

context: <instance>            "*" (any live instance) | InstanceId string form.
                                Used CLIENT-SIDE ONLY to pick which peer to dial (via
                                InstanceDiscoveryService/SRV). Never appears in the WS URL itself —
                                the URL is fixed (wss://host/cluster/v1) for every instance.
components[0]: <deployment>    "*" (merge-scan all active deployments on the instance) | DeploymentId
                                (a deterministic hash of the deployment's own id, mirroring the
                                existing ApplicationId.forUniqueName(...) pattern). Resolves via an
                                O(1) lookup into that instance's activeDeployments map when explicit.
components[1]: <element>       element name, optionally suffixed "[n]" to break ties when multiple
                                elements share a name (n = 0-based ordinal)
components[2]: <service>       fully-qualified service interface name, optionally suffixed "#<name>"
                                for a named service export (reuses ElementServiceKey's encoding)

Known limitation: transient deployments (no stable, persisted id) cannot be addressed this way — DeploymentId assumes a durable identifier that survives across lookups. Deliberate scope boundary, not a gap to close later.

Examples (as Path string form):

• abc1234://dep567/com.example.myelement/com.example.myservice#foo — specific instance, specific deployment, specific element, named service export.
• *://*/com.example.myelement/com.example.myservice — any instance, any deployment (merge-scan), unnamed service export.

4.2 Verified prior art and where the proposal needs to adapt

Concept	Proposed	What exists today	File
URI-shaped address (scheme://a/b/c, wildcard authority)	Reuse unmodified, wrap in new ElementAddress handle	Already exists as Path: CONTEXT_SEPARATOR = "://", PATH_SEPARATOR = "/", WILDCARD = "*", getNodeId() parses the authority via NodeId.nodeIdFromString(...). Path is final — cannot be subclassed, confirming composition (wrap, don't extend) is the only option	sdk-cluster/src/main/java/dev/getelements/elements/sdk/cluster/path/Path.java
Instance address round-trips as URI authority	abc1234://...	Confirmed: InstanceId.asString()/toString() produces a compact encoded token (the same mechanism NodeId.asString() uses via v1CompoundId.asEncodedString(...), minus the ApplicationId half)	sdk-cluster/.../sdk/cluster/id/InstanceId.java
Deployment-scoped lookup key (DeploymentId)	Hash of the deployment's own id, used as an O(1) lookup key	Pattern already exists for the (now-obsolete) ApplicationId: ApplicationId.forUniqueName(application.getId()) is a deterministic UUID-from-name derivation, reused verbatim for the deployment id instead. StandardElementRuntimeService.activeDeployments is already keyed by the deployment's own DB id string, so hashing that consistently gives O(1) lookup	app-node/.../Worker.java, WorkerInstanceModule.java, Context.java (existing derivation precedent); deployment-jetty/.../StandardElementRuntimeService.java (activeDeployments map)
Ordinal tie-break for same-named elements (element[2])	Embed ordinal in the path segment	Already modeled, but as a separate structured field, not a string suffix: ElementQuery(ElementRegistry, String name, int index) does registry.find(name).skip(index).findFirst()	sdk/.../query/ElementQuery.java
Fully-qualified service name + optional service name	<service>#<name> segment	ElementService has no name() of its own — the FQ name is ElementService.value().getName() (the interface's binary class name); the optional name is ElementServiceExport.name(). These two are already combined by ElementServiceKey, joined with # (SERVICE_NAME_SEPARATOR)	sdk/.../annotation/ElementService.java, ElementServiceExport.java; sdk/.../record/ElementServiceKey.java
Class<?> → remote-routed proxy	New proxy factory in ClusterElementLocator	Already exists: ProxyBuilder + RemoteInvocationHandlerBuilder, scanning @RemotelyInvokable methods, using Proxy.newProxyInstance, routing through RemoteInvoker/RemoteInvocationDispatcher	rt-server-cluster/.../ProxyBuilder.java, RemoteInvocationHandlerBuilder.java

4.3 Open design decisions

1. Reuse sdk-cluster's Path, or define a new type in sdk? RESOLVED, see §4.1. Path is reused completely unmodified (and can't be subclassed regardless — it's final). A new ElementAddress handle type wraps a validated Path via composition, not inheritance. Both ElementAddress and ClusterElementLocator live in sdk-cluster, not sdk — resolving the module-dependency question by simply not putting this API in sdk at all. Plugins wanting remote-element calls depend on sdk-cluster directly.

2. Segment separator for service + optional name. The proposal's <service>#<name> (adjusted from the original /-delimited draft during review) should reuse ElementServiceKey.toString()/tryParse directly rather than hand-rolling a second encoding of the same concept. Confirm ElementServiceKey is public/stable enough for sdk-level reuse, or whether it needs to move.

3. Ordinal parsing. The [n] suffix needs a small parse step that splits element[n] into (name, index) and constructs an ElementQuery, rather than reinventing tie-break resolution. Straightforward, but should be written against ElementQuery explicitly so the two don't drift.

4. @RemotelyInvokable vs. ElementService. ProxyBuilder's proxy generation scans methods annotated @RemotelyInvokable. It is not yet confirmed whether service interfaces annotated with @ElementService are expected to also carry @RemotelyInvokable per-method (opt-in per method), or whether exposing a service via ElementService should make all its methods remotely invokable by default. This changes what ClusterElementLocator.getRemote(ElementAddress, Class<T>) can assume about the Class<T> it's handed, and needs to be settled before the proxy factory is written.

5. Server-side symmetry. ClusterElementLocator only specifies the client-side half (construct an address, get a proxy). The server side (WsRtEndpoint in §3) needs to parse the same address format server-side to resolve ElementRegistry scope → Element → service instance. The two need to share a parser, not reimplement the grammar twice.

---

5. Phased plan

Extends WEBSOCKETS.MD §6's phases with the dispatch/addressing work from §3–4. Each phase should end green.

Phase 0: Shared safety net

Same as JEROMQ.md/WEBSOCKETS.MD Phase 0: test timeouts on every JeroMQ-touching test, socket-count leak assertions. Sunk cost regardless of downstream decisions.

Phase 1: WS transport prototype (transport only, existing dispatch)

• cluster-ws / cluster-ws-guice stubs; WsRemoteInvoker + WsRtEndpoint for one synchronous invocation, no auth, no multiplexing.
• WS API layer (resolved, see §6 Q1 — revised after Q8): a single, static WS endpoint per instance (wss://host/cluster/v1), identical for every instance and registered once at server bootstrap — never added/removed at runtime, since NodeId/ApplicationId-based per-node routing is obsolete (§6 Q8). Not registered via the existing JakartaWebsocketLoader (that mechanism scans per Element deployment; this endpoint isn't tied to any deployment and must exist regardless of what's deployed). Module placement: registered explicitly at bootstrap in deployment-jetty (the only module where ServletContextHandler/JakartaWebSocketServletContainerInitializer wiring already exists, alongside the current loader classes), not in jetty-ws itself. jetty-ws only needs a small addition — analogous to ElementsServletContextModule's existing JakartaWebsocketLoader.HANDLER_SEQUENCE wiring — to fold the resulting handler into its Handler tree.
• Round-trip against the existing deprecated dispatchers (ServiceLocatorLocalInvocationDispatcher et al.) — do not couple this phase to the ElementRegistry migration (see §3.3).
• Note: PayloadReader/PayloadWriter are stream-based (InputStream/OutputStream), not ByteBuffer-based. Framing over WS binary messages needs a stream-adapter shim; budget small time for this, WEBSOCKETS.MD undercounts it.
• Decision point (per WEBSOCKETS.MD §6 Phase 1): if the round-trip lands cleanly, continue with WS and freeze JEROMQ.md Phases 2–7.

Phase 2: Invocation semantics parity

Multiplexing by request id, async/CompletionStage invocations, cancellation, per-call timeouts, per-peer Session cache with lazy dial + concurrent-dial coalescing + idle eviction. No outbox, no reconnect loop (see WEBSOCKETS.MD §4.1 for why buffering is actively wrong for this deployment).

Phase 3: ElementRegistry-based dispatch

• Build the new LocalInvocationDispatcher implementation resolving through ElementRegistry.
• Implement DeploymentId (hash of the deployment's own id) and the O(1) activeDeployments lookup, plus the wildcard merge-scan fallback across ElementRuntimeService.getActiveRuntimes() — see §6 Q8.
• This is the first phase where WsRtEndpoint needs to do a real resolution step before dispatch(...) — see §3.2.
• Land this against the now-working Phase 1/2 transport, not the JeroMQ one, to avoid maintaining two dispatch integrations.

Phase 4: ClusterElementLocator + addressing

• Resolve the open decisions in §4.3 (Path type ownership, separator convention, @RemotelyInvokable semantics).
• Implement the address parser once, shared by the client proxy factory and the server-side resolution added in Phase 3.
• Implement ClusterElementLocator.getRemote(ElementAddress, Class<T>) on top of the existing ProxyBuilder/RemoteInvocationHandlerBuilder machinery.

Phase 5: Auth and security

Resolved, see §6 Q2/§6.1.2 — simpler than originally scoped, since auth isn't part of the cluster-ws protocol at all. The WS upgrade request carries the existing SessionSecretHeader, validated by the existing HttpServletSessionIdAuthenticationFilter/SessionService/SessionDao chain against the shared MongoDB — no new AuthScheme, no WsRtAuth bridge, no mTLS/PKI/shared-secret work. Two things to build:

• Forward the caller's session secret on user-initiated cross-instance invocations, so the existing Anonymous/User/Superuser/UNSCOPED pipeline (driven off User.Level) works unmodified.
• Provision each instance a system/service User account at deploy time, minted a normal Session via SessionDao, for pure control-plane calls (health/discovery) that have no human caller behind them.

Phase 6: Discovery + cluster topology

Plug StaticInstanceDiscoveryServiceModule into WsRemoteInvokerRegistry for tests; adapt the already-existing SRV-backed implementations (SpotifySrvInstanceDiscoveryService, JndiSrvInstanceDiscoveryService — confirmed real, not aspirational) to emit WS URLs.

Phase 7: Coexistence + cutover

elements.rt.transport config key, default jeromq for one release then ws; both transports exercised in CI; deprecation warning on jeromq selection.

Phase 8: Retire JeroMQ

Remove the six rt-*-jeromq/rt-server-cluster*-jeromq* modules, drop <jeromq.version>, delete JEROMQ_* config defaults. Keep JEROMQ.md for history.

Phase 9: Observability + tuning

Per-peer connection metrics, idle-timeout/ping-interval tuning, protocol documentation for /cluster/v1.

Phase 10: Retire app-node and clean up dead code

Deliberately last, not bundled into Phase 1. app-node (ApplicationNode/ApplicationNodeMain) is a separate executable from jetty-ws today, confirmed largely dead/broken (per direct confirmation: "pretty much dead code right now... known to be broken for a long time"). Once the WS transport and ElementRegistry dispatch are proven inside jetty-ws (Phases 1–6), fold cluster-node hosting into the jetty-ws process and delete app-node:

• Add an operational-mode CLI option to ElementsMain, following the existing --web-service idiom (jopt-simple, typed arg — that flag is enum-typed against ElementsWebServiceComponent, not raw strings, which is the established pattern here). Mode constants/type live in or near ElementsWebServices.
• Open, deferred to this phase: whether mode selection is plain String constants on ElementsWebServices (e.g. NODE_MODE_MASTER/NODE_MODE_WORKER/NODE_MODE_BOTH/NODE_MODE_NONE) or a NodeOperationalMode enum matching the ElementsWebServiceComponent pattern exactly. Decide when this phase starts, not now.
• ElementsWebServices.start() conditionally installs master-node behavior and/or stands up worker Nodes based on the resolved mode, replacing ApplicationNode's current unconditional install-both-always approach.
• Salvage only what's still useful from MasterNodeModule/WorkerInstanceModule (both currently installed unconditionally, no mode gating) — do not port app-node's dead/broken code forward wholesale.
• Delete app-node, ApplicationNodeMain, and the appnode.java thin entrypoint once jetty-ws covers its role.

---

6. Consolidated open questions

Carried forward from WEBSOCKETS.MD §7, plus new ones from this document. None of these block Phase 0; most block Phase 3 or later as noted. Each is resolved below with a working decision (§6.1) so implementation can proceed; treat these as the plan's current answer, revisit if new information contradicts them.

#	Question	Blocks	Source	Resolution
1	WS endpoint registration: jakarta.websocket annotation scanning vs. Jetty-native programmatic registration for dynamic per-node mounting	Phase 1	New (verification); revised after Q8	SIGNED OFF (revised). Single static WS endpoint per instance (wss://host/cluster/v1), registered once at bootstrap — dynamic per-node registration turned out to be unnecessary once NodeId/ApplicationId-based routing was dropped (Q8). Registered explicitly in deployment-jetty at startup, independent of JakartaWebsocketLoader's per-deployment scan. app-node retirement/mode-selection still split into Phase 10. See §6.1.1
2	Auth model for service-to-service calls: per-instance identity, shared secret, mTLS, or OIDC client-credentials	Phase 5	WEBSOCKETS.MD §7, confirmed unsolved by existing *SchemeConfiguration classes	SIGNED OFF. Not part of the cluster-ws protocol at all — out-of-band via HTTP header (SessionSecretHeader) on the WS upgrade handshake, validated through the existing SessionDao/SessionService chain against the shared MongoDB. No mTLS/shared-secret/PKI scheme. See §6.1.2
3	Invocation-cancellation semantics: does the server stop work on cancel, or only stop sending the response? Mid-frame cancel?	Phase 2	WEBSOCKETS.MD §7	SIGNED OFF. Best-effort interrupt server-side; at-most-once delivery guaranteed client-side. See §6.1.3
4	Graceful drain: how long does shutdown wait for in-flight invocations before force-closing the Session? (Note: shutdown now owned by ElementsWebServices, not ApplicationNode — see Phase 10)	Phase 2	WEBSOCKETS.MD §7	SIGNED OFF. Configurable, default 30s, adjustable per-deployment (e.g. matched to K8s terminationGracePeriodSeconds where that differs from the default). See §6.1.4
5	Max per-frame payload size and rejection behavior at the boundary	Phase 2	WEBSOCKETS.MD §7	SIGNED OFF. Default 10 MiB cap, reject with a typed PayloadTooLargeException. See §6.1.5
6	Per-peer connection density at scale (100s of peers per instance) — negligible today, needs a documented threshold	Phase 6+	WEBSOCKETS.MD §4.3, §7	SIGNED OFF. Not a near-term concern; revisit at ~100 concurrent peer Sessions per instance. See §6.1.6
7	Does the WS RT transport overlap with rpc-api/rpc-api-jetty (JSON-RPC-over-HTTP, /jrpc/invoke)? Should they merge?	Design-level, any phase	WEBSOCKETS.MD §7	SIGNED OFF. No merge, not even considered — cluster invocation semantics (multiple async result callbacks per call, via RemoteInvoker.invokeAsync) have no JRPC equivalent; JRPC's single-request/single-response HTTP model can't represent them. Fully separate, unrelated to this plan. See §6.1.7
8	ElementRegistry resolution scope: root vs. per-ApplicationId ElementScopedElementRegistry	Phase 3	New (this document)	SIGNED OFF (revised). ApplicationId is confirmed legacy, superseded by Path-based element addressing — dropped entirely. Resolution instead uses a new DeploymentId (hash of the deployment's own id, mirroring ApplicationId.forUniqueName(...)) as an O(1) key into activeDeployments, with a merge-scan fallback across getActiveRuntimes() when wildcarded. Transient deployments are not addressable. See §6.1.8
9	Path type ownership: reuse sdk-cluster's Path (domain overload with Resource/Handler addressing) vs. new sdk-level type vs. generalize Path	Phase 4	New (this document)	SIGNED OFF (revised). Reuse Path completely unmodified (it's final — can't be subclassed anyway); wrap it in a new ElementAddress handle via composition. ElementAddress/ClusterElementLocator both live in sdk-cluster, not sdk — resolves the module question by not putting this API in sdk at all. See §4.1, §6.1.9
10	Does @ElementService imply all methods are remotely invokable, or must each method opt in via @RemotelyInvokable?	Phase 4	New (this document)	SIGNED OFF. Explicit opt-in via @RemotelyInvokable per method (default-deny). See §6.1.10
11	Should the service+name path segment reuse ElementServiceKey.toString()/tryParse directly, and is that type stable enough for sdk-level reuse?	Phase 4	New (this document)	SIGNED OFF. Reuse ElementServiceKey's #-separated encoding directly; ElementAddress holds a parsed ElementServiceKey as a typed field, not a raw string. See §6.1.11

6.1 Rationale

6.1.1 — WS endpoint registration. SIGNED OFF (revised). Original answer (Jetty-native dynamic per-node registration) assumed per-node endpoints come and go at runtime, tied to NodeId's ApplicationId component. Once §6.1.8 established ApplicationId is legacy and dropped from addressing entirely, that dynamism requirement disappeared: there is now exactly one static WS endpoint per instance (wss://host/cluster/v1), identical for every instance, registered once at bootstrap and never added or removed at runtime.

Since it never changes shape at runtime, the original objection to jakarta.websocket annotation scanning (can't handle endpoints appearing/disappearing) no longer applies in principle. But the existing JakartaWebsocketLoader mechanism still isn't the right fit for a different reason: it scans per Element deployment (an endpoint's lifecycle is tied to a deployment loading/unloading), and cluster-ws's endpoint is a core platform feature that must exist regardless of what's deployed — it has no deployment to be scoped to. So: register it explicitly and unconditionally at server bootstrap, independent of the per-deployment scan, using either Jetty-native or annotation-based registration (the choice no longer matters much given it's a one-time, static registration — Jetty-native keeps it consistent with how the registration code is wired, since it lives beside the deployment-scoped loader machinery either way).

Module placement (confirmed against code, unchanged): jetty-ws depends on deployment-jetty, not the reverse. jetty-ws/ElementsMain.java bootstraps the injector; the Server object itself is built by SimpleServerProvider (common-jetty), bound via jetty-ws's JettyServerModule. The existing WS mounting (JakartaWebsocketLoader) does all its ServletContextHandler configuration inside deployment-jetty; jetty-ws's ElementsServletContextModule only wires the resulting Handler.Sequence into the top-level Handler tree. So the new registration code belongs in deployment-jetty, registered directly at startup — jetty-ws gets only the equivalent small Handler-tree wiring addition.

Process topology (confirmed and decided, unchanged): app-node is a fully separate executable from jetty-ws today (own main(), own injector, own CLI flags), and is dead/broken code that's been unmaintained for a long time. Its MasterNodeModule/WorkerInstanceModule install unconditionally together with no mode selection — every app-node process is master and worker simultaneously. Decision: fold cluster-node hosting into jetty-ws, retire app-node entirely, driven by a new CLI-selected operational mode on ElementsMain/ElementsWebServices. This work is scoped to Phase 10 (last), not bundled into the Phase 1 transport work. See Phase 10 for details; the exact mode-constant design (string constants vs. an enum matching ElementsWebServiceComponent) is deliberately left open until that phase starts.

6.1.2 — Auth model. SIGNED OFF. Corrected from the original draft: auth is not part of the cluster-ws protocol at all, and no InstanceIdentityScheme/mTLS/shared-secret/PKI mechanism is needed. It's handled entirely out-of-band, the same way every other authenticated endpoint in this codebase already works:

• The credential (SessionSecretHeader) rides the HTTP upgrade request that establishes the WS connection — an ordinary header, not a field in the cluster-ws wire protocol.
• It's validated by the existing HttpServletSessionIdAuthenticationFilter/SessionService chain, which calls SessionDao.getBySessionSecret(...) against the shared MongoDB. No new AuthScheme, no WsRtAuth bridge beyond what already exists for HTTP.
• This works because every instance already has direct database access (per the DAO-layer architecture in CLAUDE.md) — the trust boundary is "does this session secret resolve to a valid Session in the shared DB," which every node can check independently. There is no cross-instance credential to distribute, rotate, or verify; the database is the shared trust anchor already in place.
• Two call shapes, one mechanism:
  • User-initiated calls that cross instances (a REST/WS request on Instance A needs to invoke a node on Instance B on behalf of an authenticated caller): forward the real caller's session secret. The existing Anonymous/User/Superuser/UNSCOPED access-level pipeline works unmodified, since it's driven off User.Level resolved from that same Session.
  • Pure control-plane calls with no human behind them (health/discovery refresh, SimpleRemoteInvokerRegistry.refresh()'s metadata calls): each instance is provisioned a real system/service User account at deploy time and mints itself a normal Session/session-secret via SessionDao, exactly like an end user would. Session mandates a @NotNull User (confirmed in code — there's no null-user session), so this reuses the existing model rather than special-casing it or inventing a synthetic principal type.
• OIDC client-credentials is still rejected, now for a stronger reason than "extra round-trip": it would duplicate a trust check the database already performs for free.

6.1.3 — Cancellation. SIGNED OFF. Treat cancel as a race, not a hard guarantee: the server attempts Thread.interrupt() on the dispatch thread if the target service method is interruption-aware, but the invariant that matters is the caller never acts on two outcomes for one request. Once a cancel is acknowledged, suppress any result frame that hasn't already been flushed; if a result frame was already in flight when the cancel arrived, let it deliver and treat the cancel as a no-op on the client side (last-write-wins is acceptable here since it's the same race JeroMQAsyncOperation.cancel() already has today, per JEROMQ.md §7).

6.1.4 — Graceful drain. SIGNED OFF. Configurable via elements.rt.ws.drain.timeout.sec, default 30s — chosen to match Kubernetes' default terminationGracePeriodSeconds out of the box, but deployments with a different grace period should set this explicitly so the drain window never outlives the pod's own grace period (which would get the process killed mid-drain regardless). On drain start: stop accepting new inbound invocations (fail fast with a retryable error so RemoteInvokerRegistry reroutes to a healthy peer), allow in-flight invocations to complete, force-close any remaining Sessions at the timeout.

6.1.5 — Frame size cap. SIGNED OFF. Default 10 MiB (elements.rt.ws.max.frame.size), rejected at the boundary with a typed PayloadTooLargeException surfaced to the caller as a RemoteInvocationException, using WS close code 1009 at the transport level. This replaces today's silent misbehavior at the ZMQ high-water mark with an explicit, catchable failure.

6.1.6 — Connection density. SIGNED OFF. No action needed now — current cluster sizes (single- to low-double-digit instances, per WEBSOCKETS.MD §4.3) make this a non-issue. Document ~100 concurrent peer Sessions per instance as the trigger to revisit pooling/eviction tuning (the idle-eviction work is already planned in Phase 2 regardless).

6.1.7 — Overlap with rpc-api/rpc-api-jetty. SIGNED OFF. Corrected from the original draft: this isn't a "different audiences, maybe share some code" situation — it's a hard mechanical incompatibility, so there's nothing to merge or even partially share.

rpc-api's JsonRpcResource is a real, already-live JAX-RS endpoint (POST /jrpc/invoke, GET /jrpc/manifest, pulled into jetty-ws via rpc-api-jetty) that resolves calls through LocalInvocationDispatcher (bound to ServiceLocatorLocalInvocationDispatcher in JsonRpcModule) — the same dispatcher abstraction the cluster RPC stack uses. That's a real code-level connection, not just a naming coincidence. But cluster invocations support multiple asynchronous result callbacks per call (RemoteInvoker.invokeAsync(Invocation, List<Consumer<InvocationResult>>, InvocationErrorConsumer)), which is fundamentally incompatible with JRPC's single-request/single-response HTTP model — there's no way to express "this one call produces several results over time" in a JSON-RPC-2.0-over-HTTP exchange. cluster-ws and rpc-api/rpc-api-jetty are unrelated, stay unrelated, and this plan does not touch JRPC in any way.

6.1.8 — ElementRegistry scope. SIGNED OFF (revised). Original answer (per-ApplicationId ElementScopedElementRegistry) was wrong on two counts:

1. ElementScopedElementRegistry isn't scoped to an Application at all — it's a thin per-Element delegating wrapper (parent, one Element), constructed one-per-Element-instance (SharedElement.java:45, GuiceElementLoader.java:47), that just forwards stream()/publish()/onEvent() to its parent. It has no notion of ApplicationId.
2. ApplicationId itself is confirmed legacy — a holdover from an older system, superseded by Path-based element addressing. It's dropped from this design entirely, not just from this one lookup.

Actual mechanics: StandardElementRuntimeService.doLoadDeployment(...) creates one RootElementRegistry.newSubordinateRegistry() per deployment, stored in activeDeployments keyed by the deployment's own DB id (a plain String). Confirmed: RootElementRegistry.stream() does not aggregate downward into subordinates — stream() concatenates only its own loaded list with its parent's stream, never a child's. Each deployment's registry is fully isolated from its siblings. So there's no existing flattened, instance-wide view to query.

Resolution: a new DeploymentId — a deterministic hash of the deployment's own id, reusing the exact derivation pattern already established for ApplicationId.forUniqueName(application.getId()) (Worker.java, WorkerInstanceModule.java, Context.java), just applied to the concept that's actually load-bearing now. A Path carrying an explicit DeploymentId resolves via an O(1) lookup into activeDeployments, then ElementQuery(registry, elementName, ordinal) within that one registry. A wildcarded DeploymentId (*) falls back to merging Stream<Element> across every RuntimeRecord from ElementRuntimeService.getActiveRuntimes() and applying the same name+ordinal resolution — an O(n) scan, acceptable at current cluster/deployment counts (same reasoning as §6.1.6).

Known limitation: transient deployments (no stable, persisted id) aren't addressable through this scheme — DeploymentId assumes a durable identifier that survives across lookups. Deliberate scope boundary of the addressing design, not an oversight.

6.1.9 — Path ownership. SIGNED OFF (revised). Original answer (new narrower type in sdk, sharing parsing internals with Path privately) is superseded — checking Path's actual source changed the shape of the decision entirely. Path is declared public final class Path, so subclassing was never on the table as a real option to begin with. And sdk genuinely cannot reference sdk-cluster types at all: confirmed via the actual pom.xml files that sdk-cluster already depends on sdk (not the other way around), so sdk doing the reverse would be a circular module dependency, not just an undesirable new one.

Resolution: Path is reused completely unmodified — no subclass, no new methods, no domain overload, since we're not touching the type at all. A new ElementAddress handle type is constructed from a Path via a validating factory (parses/checks components once at construction, fails fast on malformed input), used by composition rather than inheritance. Both ElementAddress and ClusterElementLocator (renamed twice: RemoteElementRegistry → ClusterElementRegistry → ClusterElementLocator, since "Registry" implied ElementRegistry's enumeration contract, which this type doesn't have) live in sdk-cluster alongside Path/InstanceId — not in sdk. This sidesteps the circular-dependency question entirely rather than solving it: the API just doesn't live in sdk. Plugins wanting remote-element calls add sdk-cluster as a dependency alongside sdk.

6.1.10 — @RemotelyInvokable opt-in. SIGNED OFF. Require explicit per-method @RemotelyInvokable, consistent with ProxyBuilder's existing scan behavior and a default-deny security posture. Making every @ElementService method automatically network-reachable would silently expose methods element authors never intended to be callable cross-instance. Document this requirement as part of the ClusterElementLocator contract so it isn't a surprise the first time a call 404s.

6.1.11 — Service+name segment encoding. SIGNED OFF. Reuse ElementServiceKey's existing #-separated encoding — the #name suffix is simply absent for an unnamed service, matching ElementServiceKey's own defaulting. Rather than just matching the format, ElementAddress holds a real, parsed ElementServiceKey as one of its fields: ElementAddress.from(Path) parses the final path component through ElementServiceKey's own parse logic (not a hand-rolled re-implementation of the same #-split), so ElementAddress exposes something like getServiceKey(): ElementServiceKey rather than a raw string. This is the same principle as the rest of ElementAddress's design (§4.1) — validate and resolve into concrete typed fields once at construction, not re-parse strings at every use site.

Refinement (ElementServiceKey, not ElementAddress): ElementServiceKey<ServiceT>(Class<ServiceT> type, String name) is a record whose type field is an already-resolved Class<ServiceT>, not a string — so embedding it in ElementAddress necessarily means resolving a Class<?> at construction time. The existing ElementServiceKey.tryParse(ElementRecord, String) can't be reused for this: it resolves via elementRecord.classLoader(), the target Element's own isolated classloader, which only exists for Elements already loaded in this JVM — a client addressing a remote Element it hasn't loaded locally has no ElementRecord to supply. Add a new overload, ElementServiceKey.tryParse(ClassLoader, String), sharing the same split/parse logic as the existing one, just sourcing the classloader differently. ElementAddress.from(Path) defaults to Thread.currentThread().getContextClassLoader(); ElementAddress.from(Path, ClassLoader) for explicit control. No separate Class<?> field is needed on ElementAddress itself — getServiceKey().type() already exposes it via the embedded record.

---

7. References

• JEROMQ.md — current JeroMQ subsystem state, issue inventory, renovation-path alternative.
• WEBSOCKETS.MD — full transport architecture, pros/cons, migration phases this document extends.
• rt-server-cluster-node/.../AbstractLocalInvocationDispatcher.java, ServiceLocatorLocalInvocationDispatcher.java, ContextLocalInvocationDispatcher.java, MasterNodeLocalInvocationDispatcher.java — current (deprecated) dispatch implementations.
• sdk/.../ElementRegistry.java, MutableElementRegistry.java; sdk-spi/.../RootElementRegistry.java, ElementScopedElementRegistry.java — target dispatch resolution.
• sdk/.../query/ElementQuery.java — existing ordinal tie-break query.
• sdk/.../annotation/ElementService.java, ElementServiceExport.java, ElementServiceImplementation.java; sdk/.../record/ElementServiceKey.java — service naming model.
• sdk-cluster/.../cluster/path/Path.java — existing URI-shaped addressing type (Resource/Handler domain).
• sdk-cluster/.../id/NodeId.java, InstanceId.java, ApplicationId.java — compound ID types. ApplicationId is confirmed legacy/dropped from this design (§6 Q8); InstanceId is the sole surviving routing identifier.
• deployment-jetty/.../StandardElementRuntimeService.java — per-deployment subordinate registry creation (doLoadDeployment), activeDeployments map (keyed by deployment id).
• sdk-deployment/.../ElementRuntimeService.java — getActiveRuntimes()/RuntimeRecord, the fallback merge-scan path for wildcarded DeploymentId lookups.
• sdk-guice/.../RootElementRegistryModule.java — root MutableElementRegistry eager-singleton binding.
• app-node/.../Worker.java, WorkerInstanceModule.java, rt-server-cluster-node/.../Context.java — existing ApplicationId.forUniqueName(...) derivation pattern, reused for DeploymentId.
• rt-server-cluster/.../ProxyBuilder.java, RemoteInvocationHandlerBuilder.java — existing Class<?> → remote-proxy machinery.
• rt-common/.../remote/RemoteInvoker.java, PayloadReader.java, PayloadWriter.java.
• rt-server-cluster/.../SimpleRemoteInvokerRegistry.java — health-driven pruning behavior.
• rt-server-cluster/.../InstanceDiscoveryService.java, SpotifySrvInstanceDiscoveryService.java, JndiSrvInstanceDiscoveryService.java.
• sdk-dao/.../SessionDao.java; sdk-model/.../session/Session.java, SessionCreation.java; sdk-model/.../user/User.java — auth model for cluster RPC (§6 Q2).
• common-util/.../security/SessionSecretHeader.java; common-servlet/.../security/HttpServletSessionIdAuthenticationFilter.java, HttpServletAuthenticationFilter.java — existing session-token validation pipeline, reused as-is for the WS upgrade handshake.
• sdk-test-element-ws/.../EchoEndpoint.java, deployment-jetty/.../loader/JakartaWebsocketLoader.java — existing WS endpoint precedent.
• jetty-ws/.../ElementsMain.java, ElementsWebServices.java, common-jetty/.../SimpleServerProvider.java — jetty-ws process bootstrap and Server construction; target home for Phase 10's operational-mode CLI flag.
• app-node/.../ApplicationNode.java, ApplicationNodeMain.java — current (separate, dead/broken) cluster-node executable; retired in Phase 10.
• app-node/.../guice/MasterNodeModule.java, WorkerInstanceModule.java — Guice modules to selectively salvage into jetty-ws in Phase 10.
• app-node/.../ApplicationNode.java — current unconditional JeroMQ module installation.

[ptwohig]

Cluster RPC: Restoration Plan

Status: proposal for team review.
TL;DR: the cluster RPC fabric (the rt-* modules) was deliberately disabled to ship Elements 3.0 because there wasn't a strong enough business case to keep it on the release path. Upcoming product requirements now need it back. This document lays out the two paths back to a working fabric (renovate the JeroMQ stack or replace it with Jetty WebSocket) and recommends a parallel probe that picks the winner in about one engineering week.

Recommendation: spend a week running test-suite cleanup + WebSocket prototype + JeroMQ definite-bug fixes in parallel; commit to WebSocket if the prototype clears its round-trip; fall back to JeroMQ renovation if it doesn't. Separately, replace the opaque quality: double capacity score with concrete available / total resource accounting (transport-independent; do this regardless).

Supporting documents (this doc is a synthesis):

• JEROMQ.md: full diagnosis of the existing JeroMQ stack and 8-phase renovation plan.
• WEBSOCKETS.MD: Jetty WebSocket replacement plan.
• RT-TRANSPORT-COMPARISON.md: side-by-side comparison with risk and effort estimates.
• CLUSTER-CAPACITY.md: capacity reporting redesign.

---

1. Context

The cluster RPC fabric is the system that lets Elements instances call each other. RemoteInvoker dispatches, InstanceConnectionService discovers peers, SimpleRemoteInvokerRegistry picks candidates by health and capacity, Nodes host the actual work. It is built on JeroMQ today (DEALER/ROUTER sockets, a custom async-connection abstraction, an in-process ZMQ control protocol).

It was working. The JeroMQEndToEndIntegrationTest once exercised the full pipeline (sync invocations, async invocations, futures, custom consumer callbacks, deliberate-error echo scenarios) and passed. It was deliberately stood down to ship Elements 3.0 on schedule when there wasn't a strong business case for keeping it actively maintained. Every test method now carries @Test(enabled = false); that's regression from a working state, not greenfield.

It's coming back. Restoration is on the roadmap because upcoming requirements need it. This document exists to choose the path back, not to debate whether we want one.

What this means for risk: the invocation pipeline above the transport (serialization via PayloadReader/PayloadWriter, dispatch via RemoteInvoker and LocalInvocationDispatcher, routing through SimpleRemoteInvokerRegistry, node lifecycle, the SDK contracts in general) has been validated in anger. The unknown is concentrated in the bottom layer. The JeroMQ transport, its custom thread model, and the in-process control protocol. That's where the bit-rot is. That's the part that has to change.

---

2. The choice

Two paths to a working cluster RPC fabric:

	Renovate	Replace
What	Fix the JeroMQ stack in place. Rewrite the threading model. Bump JeroMQ 0.5.2 → 0.6.x.	Build cluster-ws on Jetty's WebSocket API alongside the JeroMQ modules. Cut over. Retire rt-*-jeromq after a deprecation window.
Detail	JEROMQ.md §8 (Phases 0–7)	WEBSOCKETS.MD §6 (Phases 0–7)
Effort	~30–40 eng-days	~25–35 eng-days
Critical-path risk	Phase 2 (thread-model rewrite); Phase 4 (0.6.0 upgrade hang chase)	Phase 1 (does the prototype work?); Phase 2 (cancellation semantics)

The two plans share Phase 0. Test timeouts, leak assertions, and the definite-bug fixes the JeroMQ stack needs regardless. Phase 0 is a sunk cost we want either way.

---

3. The framing that makes the decision easy

In the way we actually use it, JeroMQ is a TCP socket with length-prefixed framing and an opaque-bytestring sender identity. We use DEALER and ROUTER for RPC, PUSH/PULL on inproc:// for one in-process hop, multipart ZMsg for envelopes, and CURVE for encryption. That's it.

We don't use the things JeroMQ is actually known for: PUB/SUB topic filtering, DEALER multi-peer fanout/load-balancing, REQ/REP, PAIR sockets, inproc-as-cross-process IPC, send-while-disconnected buffering, implicit transport-level reconnect.

So the right question isn't "JeroMQ vs WebSocket as competing transports." It's: given that we use ZMQ as a glorified TCP socket with framing, what does WebSocket give us that ZMQ doesn't, scoped to our usage?

What ZMQ-as-we-use-it does that WS doesn't

Feature we use	WS equivalent
Length-prefixed framing	WS frames are length-prefixed. Free.
Multipart ZMsg	One binary frame with a small header struct + payload. Same info, flatter layout.
ROUTER identity envelope	The Session reference. Strictly better. A typed Java handle vs an opaque byte string.
inproc:// between node ingress and dispatch	BlockingQueue / CompletableFuture. We were paying for a socket where a reference would do.
CURVE encryption	TLS via Jetty's existing SSL config. CURVE PKI retires.
ZMonitor event log	Jetty WS listener callbacks + access log.

Net: essentially nothing. Every ZMQ feature we touch has a direct WS analogue, several of them strictly simpler.

What WS does that ZMQ doesn't (scoped to us)

WS feature	What it buys us
HTTP upgrade handshake	Path-based dispatch (/cluster/v1/instance/<id> vs /cluster/v1/node/<id>). Replaces the in-instance JeroMQMultiplexRouter + JeroMQDemultiplexRouter + JeroMQCommandServer + the whole control-plane protocol on top of them
HTTP headers	Auth attaches naturally: Authorization, OIDC bearer tokens (via the existing DefaultOidcSchemeConfiguration), session tokens, X-Elements-* scope headers. Platform auth becomes transport auth
TLS via existing Jetty SSL	One PKI to manage. Same certs that front REST
Standard close codes	1000 normal, 1001 going away, 1011 internal error, etc. ZMQ drops are TCP RST/FIN with no semantic
L7 proxy / mesh compatibility	K8s ingresses, load balancers, Istio/Linkerd parse HTTP. They don't parse ZMQ. RT traffic gets the same routing/observability/mTLS as the REST API
Ping/pong control frames	First-class liveness check; JeroMQMonitorThread goes away
Standard observability	Access log, Jetty metrics, W3C trace-context headers. All work for WS, none for ZMQ
Pure-Java threading	Plain java.util.concurrent; no libzmq mutex fighting Java's Lock/Semaphore/Condition (see §4 below)

The honest framing is: in our usage, WebSocket is a strict superset of JeroMQ, plus an HTTP layer we'd otherwise have to ignore.

---

4. Why the JeroMQ stack is hard to fix in place

A short version of JEROMQ.md §6 (13 documented issues. These are the ones that matter most):

1. JeroMQAsyncConnectionPool.doClose() has an inverted ternary (line 224). The pool never actually closes its connections on shutdown. Sockets leak every teardown. Definite bug, easy fix.

2. 0.5.2 is pinned; 0.6.0 hangs all connections. Upstream 0.6.0 added explicit ZContext mutex locks. Our pervasive use of ZContext.shadow() + per-call short-lived clients + arbitrary-thread socket.close() deadlocks under the new locking.

3. The threading model fights Java concurrency primitives. This is the structural one. JeroMQ's design assumption is that network I/O (via inproc:// sockets) is the synchronization primitive. We instead use ReentrantLock + Condition, CountDownLatch, Semaphore(4096) + AtomicReferenceArray<Runnable> over a NIO Pipe, ExecutorService, and Exchanger<Exception> (all bridged to libzmq via a hand-rolled command bus (JeroMQAsyncThreadContext). Every bridge is a place a libzmq mutex can deadlock against a Java monitor. The Thread.interrupt() problem (upstream #116, #283, #347) is open in 0.6.0) Context.term() must run before any interrupt, but we interrupt freely. This is what CLAUDE.md flags as the "threading nightmare."

4. Tests are disabled wholesale. JeroMQEndToEndIntegrationTest has @Test(enabled = false) on every method. The instance-connection integration test has a subscribeToDisconnect copy-paste bug (registered on first twice instead of first + second) that hangs the suite.

5. Each JeroMQNode allocates an unbounded cached thread pool. No central concurrency limit. shutdownNow().awaitTermination(10, MINUTES) is the timeout.

6. JeroMQRoutingServer.close() closes the poller and shadow context before joining the monitor thread. Race-prone on 0.5.2; hangs on 0.6.0.

7. org.zeromq:jeromq leaks into rt-server-cluster-node and rt-server-testkit (which are meant to be transport-agnostic).

The renovation plan can fix all of these, but #3 is structural. JeroMQ doesn't want to be used with Java concurrency primitives, and we use them everywhere. Phase 2 of the renovation is essentially "rewrite the threading model to lean into JeroMQ's inproc:// philosophy." It's the hardest phase and the one most likely to overrun.

---

5. Why WebSocket dramatically simplifies the architecture

Beyond the trade-off table in §3, the WebSocket replacement deletes entire subsystems rather than reimplementing them:

The in-instance control plane disappears. The control client/server protocol (JeroMQControlClient, JeroMQAsyncControlClient, JeroMQCommandServer, JeroMQRoutingCommand, JeroMQControlResponseCode, JeroMQRoutingStatus) exists because ZMQ gave us one socket bound to the public address and we had to layer command/opcode dispatch on top to handle multiple operation kinds (getInstanceStatus, openRouteToNode, closeRoutesViaInstance, plus actual invocation forwarding). WebSocket + URL paths makes that workaround unnecessary at the protocol level.

Current control-plane call	WS equivalent
client.getInstanceStatus()	unneeded. Caller has the NodeId already (it encodes the instance)
client.openRouteToNode(nodeId, addr)	the connection is the route: wss://host/cluster/v1/node/<nodeId>
client.closeRoutesViaInstance(...)	close the Session
client.openBinding(nodeId) (local)	wsServer.registerEndpoint("/cluster/v1/node/" + nodeId, handler)

The in-instance routers disappear. JeroMQMultiplexRouter and JeroMQDemultiplexRouter did fan-out from a single ROUTER bind to per-node backend sockets, via ZMQ identity envelopes. WS does that via URL paths: Jetty resolves the path and hands the request to the right endpoint handler.

The custom thread model disappears. Jetty's QueuedThreadPool already runs in this process. It accepts, parses, dispatches. The libzmq I/O reactor (~cores + 1 threads), the JeroMQAsyncConnectionService IO threads (another cores + 1), the per-JeroMQInstanceConnectionService server thread, the per-ROUTER monitor thread, the per-ICS scheduler thread, and the per-JeroMQNode unbounded cached pool all collapse into "Jetty handles it."

The CURVE PKI disappears. TLS via the existing Jetty SSL config. One set of certs, same as the REST API.

JeroMQInstanceConnectionService shrinks from ~680 lines to ~80. Most of its mass is choreographing the control protocol. Without that protocol, it's a thin per-peer-invoker cache plus discovery glue.

---

6. What about reliability / buffering / reconnect?

JeroMQ's DEALER sockets buffer outbound messages while a peer is unreachable and flush on connect. WS is connect-then-send. For our deployment shape this isn't a feature we lose. It's a feature we shouldn't replicate, because the application architecture already does something better.

Three layers prevent the "outbound message queued behind a dead peer" scenario:

1. Kubernetes SRV records gate on readiness. Discovery only lists pods that pass their readiness probe. If the address resolves, the receiving pod is bound and serving.

2. Dial-on-demand. No idle pre-opened connection waiting for an outage. We resolve and dial when a call is ready to send. No queue of "messages that arrived during a disconnect" because we don't connect until there are messages.

3. RMI-driven health pruning at the registry. SimpleRemoteInvokerRegistry.refresh() walks every active InstanceConnection every REFRESH_RATE_SECONDS (default 30 s) and calls getInstanceMetadataAsync via RMI. A peer that times out or errors has its nodes removed from the routing snapshot (SimpleRemoteInvokerRegistry.java:328). Unreachable peers stop receiving new work within one refresh cycle.

Buffering would actively conflict with this design. Replaying a buffered message after a reconnect would deliver stale work to a peer the registry has already pruned and re-routed away from. The application semantic is "if a peer doesn't answer, route to a different one." A transport that silently reconnects underneath would violate the snapshot contract.

What we do need, but small:

• A short connect timeout (~5 s) on the initial dial.
• A per-peer Session cache so subsequent calls reuse the open connection and amortize the handshake; idle-evict by time.
• Concurrent-call coalescing: if ten calls arrive while the first dial is in progress, they chain off the same CompletableFuture<Session>.

No outbox. No reconnect loop. No buffering policy. The deployment shape closes the gap that buffering was filling.

---

7. The capacity model side-quest

The InstanceMetadataContext.getInstanceQuality(): double score is opaque and unimplemented in production. The only getInstanceQuality impl in the tree is the integration-test mock returning ThreadLocalRandom.current().nextDouble(). The registry weights candidates by quality, but there's no defined semantics for "quality 0.7" and no value that means "do not route here."

Replace it with concrete resource accounting:

public class InstanceMetadata {
    private Set<NodeId> nodeIds;
    private int availableResources;
    private int totalResources;
}

Peers report both numbers. The registry sorts candidates by availableResources (absolute, not ratio. "do you have room?" not "what's your utilization?"). A peer at availableResources <= 0 is excluded from new work; existing connections are unaffected.

Three commitments needed:

1. Local optimistic debit. Between refreshes (30 s default), the registry tracks "I gave peer X N units since the last refresh" and subtracts from the reported available when sorting. Without this, a burst of allocations all routes to the same peer. Resets each refresh.

2. A ResourceBudget / ResourceClaim SPI. Try-with-resources pattern; elements claim units for the duration of their work; the peer aggregates open claims and reports total - sum(claims) = available. claim() throws CapacityExhaustedException when the budget is exhausted.

3. Configured total, not auto-detected. A @Named Guice binding in ApplicationNodeModuleDefaults. Explicit calibration beats fragile cgroup-reading auto-detection.

Defer to v2: per-node breakdown (different nodes on the same instance report independent budgets), soft vs hard caps, capacity-driven autoscaling.

This refactor is independent of the transport choice. It lives in rt-server-cluster, not in any transport-specific module. Do it on either path; preferably before the transport decision so production gets a real signal instead of random doubles.

Full design: CLUSTER-CAPACITY.md.

---

8. Recommendation

Run a one-week parallel probe, then commit.

Phase 0 (shared, ~2 eng-days)

Do this regardless of which transport survives:

• Add @Test(timeOut = ...) to every JeroMQ-touching test. No more silently-hanging suite.
• Add @AfterMethod baseline socket-count assertions to catch leaks immediately.
• Fix the subscribeToDisconnect copy-paste bug in JeroMQInstanceConnectionServiceIntegrationTest.

Phase 1 (parallel, ~3–5 eng-days)

In parallel:

• WebSocket prototype. Stand up cluster-ws + cluster-ws-guice stubs. Implement a minimal WsRemoteInvoker round-trip using the existing PayloadReader/PayloadWriter. One sync invocation, one response, in-JVM. Decision point at the end.
• JeroMQ definite-bug fixes. Fix JeroMQAsyncConnectionPool.doClose(). Fix the requestConnection race. Fix JeroMQRoutingServer.close() ordering. Add a timeout to the per-call JeroMQControlClient. Replace the Exchanger<Exception> startup signaling with a timed CompletableFuture. These are fixes we want regardless of which transport survives. The JeroMQ stack keeps running through the migration window.
• Start the capacity refactor (CLUSTER-CAPACITY.md Phase 0–1). Deprecate the old getInstanceQuality(), land the ResourceBudget SPI with a default impl, wire it into InstanceMetadataContext. Production gains a real capacity signal in parallel.

Decision

At the end of Phase 1:

• If the WS prototype's round-trip works end-to-end with the existing serializers, commit to WebSocket. Continue with WEBSOCKETS.MD Phases 2–7. The JeroMQ stack stays on life support through the coexistence window in Phase 5 and retires in Phase 6.
• If the WS prototype hits a structural blocker (most likely candidates: invocation-cancellation semantics, in-flight call handling during Session close, graceful shutdown ordering), continue the JeroMQ renovation. Phases 2–7 of JEROMQ.md take over.

The capacity refactor proceeds in parallel either way.

Why this is the right shape

• Phase 0 is universally useful.
• The Phase 1 prototype is a cheap probe: a week of work that answers a high-stakes question definitively.
• The Phase 1 JeroMQ bug fixes are work we want regardless, so no parallel-effort is wasted.
• Decision-cost of picking later is genuinely zero: every artifact is reused on either path.

---

9. Open questions for the team

These are decisions we'd like input on before Phase 1 lands:

• Service-to-service auth model. If we go WebSocket, what carries cluster identity? Per-instance service identities? Per-cluster shared secret? mTLS client certs? OIDC client-credentials flow? The existing user-facing token infrastructure isn't quite a fit.

• Invocation cancellation semantics. When a caller cancels a pending invocation, does the server stop executing, or just stop sending the response? What if the response is mid-frame on WS?

• Graceful drain on shutdown. When ApplicationNode.shutdown() runs, how long do we wait for in-flight invocations before force-closing? Suggest a configurable timeout, default 30 s.

• What counts as a "resource" in the capacity model. WebSocket connections? Game instances? Tenant bindings? The system is unit-agnostic; the operator calibrates the total. But we should pick a default for the game-server case and document it.

• Module renaming scope. New code lands as cluster-ws / cluster-ws-guice / cluster-node-ws. The existing rt-server-cluster-* modules can rename to cluster-* during retire-JeroMQ phase (the work is touching them anyway). rt-*-jeromq modules keep their names through retirement. Confirm this rename plan.

---

10. Naming

The rt-* prefix is historical shorthand for "real-time," which these modules don't really do. They're the cluster RPC fabric. We're using this restoration work to start migrating to clearer names:

• New modules drop rt-server-. cluster-ws, cluster-ws-guice, cluster-node-ws, cluster-node-ws-guice.
• New URL path is /cluster/v1/... rather than /rt/v1/....
• Existing rt-server-cluster-* modules rename to cluster-* during retire-JeroMQ phase, when those modules are being touched anyway.
• rt-*-jeromq modules keep their existing names through retirement to minimize rename churn during the coexistence window. They're going away, so we don't pay the rename cost.

---

11. What we're asking for

Read this doc and the four supporting docs (JEROMQ.md, WEBSOCKETS.MD, RT-TRANSPORT-COMPARISON.md, CLUSTER-CAPACITY.md). Push back on framing or recommendations you disagree with. Then we'd like alignment on:

• Phase 0 starts immediately. Two engineer-days, mostly mechanical, no transport commitment.
• Phase 1 runs in parallel: WS prototype, JeroMQ definite-bug fixes, capacity refactor start. ~3–5 engineer-days.
• We commit to one transport path at the end of Phase 1 based on the prototype outcome.
• Module/URL naming as described in §10.
• Capacity refactor proceeds independently and lands regardless.

[ptwohig]

RT Transport: Side-by-Side Comparison

Decision aid for JEROMQ.md (renovate the existing JeroMQ stack) vs WEBSOCKETS.MD (replace it with a Jetty-WebSocket transport).

Why we're picking between these now. The cluster RPC fabric was deliberately disabled for the Elements 3.0 release. Not because it was broken, but because there wasn't a strong enough business case at the time to justify maintaining it on the release path. That call is now being revisited: upcoming product requirements need the rt system restored. This document exists to choose the path back to a working fabric, not to debate whether we want one.

Both plans share Phase 0 (test-suite safety net). Diverge from Phase 1.

---

1. Phase-by-phase

Phase	JEROMQ.md §8 (Renovate)	WEBSOCKETS.MD §6 (Replace)
0	Add timeOut to JeroMQ tests; assert baseline ZMQ socket count in @After; capture thread dumps on timeout.	Same work; shared sunk cost.
1	Fix the definite bugs: JeroMQAsyncConnectionPool.doClose() ternary, requestConnection race, JeroMQRoutingServer.close() ordering, per-call JeroMQControlClient timeout, Exchanger timeout.	Stand up cluster-ws + cluster-ws-guice stubs. Implement a minimal WsRemoteInvoker round-trip with the existing PayloadReader/PayloadWriter. Decision point.
2	Rewrite the threading model: replace JeroMQAsyncThreadContext's pipe + semaphore + atomic-array bus with either (a) one IO thread per pool wired through inproc:// PAIR sockets, or (b) a single reactor thread over a NIO Selector.	Feature parity: multiplexed invocations by request id, CompletionStage-returning RPCs, invocation cancellation, per-call timeouts, outbox + reconnect-with-backoff for disconnected-peer optimistic sends.
3	Replace JeroMQAsyncConnectionPool with a boring pool (commons-pool2 or equivalent): bounded, evictable, predictable close().	Auth + security: wire /cluster/v1 into the existing Jetty auth filter chain (Authorization, OIDC via DefaultOidcSchemeConfiguration, session tokens); confirm access-level scoping; TLS via the existing SSL config.
4	Bump <jeromq.version> to 0.6.0; run the (now timeout-protected) suite under heavy contention; file upstream issues for anything that still hangs.	Discovery: adapt InstanceDiscoveryService (static + SRV) to emit WS host:port URLs; callers append /cluster/v1/instance/<id> or /cluster/v1/node/<id> from existing identifiers (no extra routing table).
5	Tidy the module graph: drop the direct org.zeromq:jeromq dep from rt-server-cluster-node and rt-server-testkit; gate JeroMQ installation in ApplicationNode behind a feature flag (elements.rt.enabled, default false).	Coexistence + cutover: elements.rt.transport=ws|jeromq|none; ship one release defaulting to jeromq, one defaulting to ws; run both transports in CI.
6	Observability: socket count, pool occupancy, queue depth, invocation latency, IO-thread liveness; /healthz contributor; retire JeroMQDebugCounter.	Retire JeroMQ: delete rt-*-jeromq modules, drop <jeromq.version>, delete the JEROMQ_* config keys. Keep JEROMQ.md for historical reference.
7	Reconsider whether the cluster mesh is still the right architecture for Elements 4.x.	Observability + tuning: per-peer connection metrics, idle-timeout / ping-interval tuning, publish the /cluster/v1 frame format.

---

2. Scoped to how we actually use ZMQ

The broad trade-off table in §3 covers JeroMQ vs WebSocket as transports in general. But this codebase doesn't use most of what ZMQ uniquely offers. A grep through the JeroMQ code confirms our usage surface is narrow:

What we use: DEALER for client RPC, ROUTER for the public bind and the per-node frontend, PUSH/PULL over inproc:// for intra-JVM node fan-out, inproc:// for the control plane, multipart ZMsg framing, ZMQ routing-identity envelopes, CURVE for transport encryption, ZMonitor for socket-event logging.

What we don't use: PUB/SUB topic filtering, DEALER multi-peer round-robin (each invoker dials one peer), REQ/REP, PAIR sockets, inproc-as-cross-process-IPC, send-while-disconnected buffering (per K8s SRV gating. See WEBSOCKETS.MD §4.1), implicit auto-reconnect, deliberate HWM-tuned backpressure.

In other words, we are using ZMQ as TCP + length-prefixed framing + routing-identity envelopes, and nothing else. Scoping the comparison to that shape changes the picture materially.

2.1 What ZMQ-as-we-use-it does that WS doesn't

ZMQ feature we use	WS equivalent	Delta
Length-prefixed framing	WS frames are length-prefixed	None. Free in both
Multipart ZMsg (N frames per logical message)	One binary frame containing a small header struct (request id + opcode + payload length) + payload	Flatter layout, same information
ROUTER identity envelope ("which conn sent this?")	The Session reference	WS strictly better. Typed Java handle vs opaque byte string
inproc:// between node ingress and dispatch	BlockingQueue / CompletableFuture; LocalInvocationDispatcher already covers cross-JVM-call-but-target-is-local	We were paying for a socket where a Java reference would do
CURVE encryption	TLS via Jetty's existing SSL config	One PKI retires
ZMonitor event log	Jetty WS listener callbacks + access log	Standard tooling

Material things ZMQ gives us that WS doesn't, in our usage: essentially nothing. The features that are arguably uniquely ZMQ (PUB/SUB, DEALER fanout, inproc-as-IPC) aren't on our usage list.

2.2 What WS does that ZMQ-as-we-use-it doesn't

WS feature	What it buys us
HTTP upgrade handshake	Path-based dispatch (/cluster/v1/instance/<id> vs /cluster/v1/node/<id>). Replaces JeroMQMultiplex/DemultiplexRouter + JeroMQCommandServer
HTTP headers	Auth attaches naturally: Authorization, OIDC, session tokens, X-Elements-* scope headers. Platform auth becomes transport auth
TLS via Jetty's SSL config	One PKI to manage. Same certs that front REST. CURVE retires
Standard close codes / reasons	WS close has a status code (1000 normal, 1001 going away, 1011 internal error, etc.) + reason. ZMQ drops are TCP RST/FIN
L7 proxy / mesh compatibility	K8s ingresses, load balancers, Istio/Linkerd parse HTTP. They don't parse ZMQ
Ping/pong control frames	First-class liveness check; JeroMQMonitorThread goes away
Standard observability	Access log, Jetty metrics, W3C trace-context headers, distributed-trace tooling. All work for WS, none for ZMQ
Permessage-deflate	Frame compression negotiated at handshake; not built into ZMQ
Browser interop (future option)	Future browser-based admin / live tooling is feasible. Impossible with ZMQ

2.3 Takeaway

The honest framing is not "JeroMQ vs WebSocket as competing transports." It is: the way we use JeroMQ, WebSocket is a strict superset, plus an HTTP layer we'd otherwise have to ignore. The HTTP envelope is the entire value delta. And it happens to be the part that integrates with every other system we already operate.

This reframes §3 below. Many of the rows there read as "we lose X feature" or "we have to build Y." Read through the §2 lens, most of those Xs are features we never used in the first place, and most of the Ys are direct simplifications (typed Session instead of opaque identity, BlockingQueue instead of inproc).

---

3. Trade-offs

Dimension	Renovated JeroMQ (JEROMQ.md)	WebSocket (WEBSOCKETS.MD)
Transport libraries	org.zeromq:jeromq 0.6.x and Jetty	Jetty only
Process thread budget	libzmq IO threads (~cores+1) + async-conn IO threads (~cores+1) + per-ICS server + per-ICS scheduler + per-ROUTER monitor + per-Node unbounded cached pool	Jetty QueuedThreadPool (already running)
Sync primitives	Lock/Semaphore/Condition/Exchanger bridged to libzmq via a hand-rolled command bus. See JEROMQ.md §5	Plain java.util.concurrent; no bridge
Auth integration	New bridge from CURVE + our PKI to access-level scoping; JeroMQSecurityProvider + PEM + JEROMQ_ALLOW_PLAIN_TRAFFIC escape hatch	Inherits the Jakarta WS / Jetty filter chain; Authorization + OIDC + session tokens work the day Phase 3 lands
PKI	CURVE keys distributed and rotated separately	Same TLS that fronts the REST API
Network observability	Opaque outside the JVM; only JeroMQMonitorThread + DEBUG logs	HTTP-adjacent: access log, mitmproxy, devtools, distributed-trace headers, service-mesh sidecars
Firewall / NAT traversal	TCP/28883 default; needs operator coordination	TCP/443 + HTTP upgrade
Backpressure model	ZMQ_SNDHWM + a Java-side Semaphore(4096) + Node's unbounded cached pool (three regimes, two of which disagree	TCP window + WS frame buffer) single stack
Send-while-disconnected	DEALER buffers and flushes on connect	Not needed and would conflict with the application's design: SimpleRemoteInvokerRegistry health-prunes unreachable peers from the routing snapshot every refresh cycle, so buffered replay would deliver stale work to a peer the registry has already routed away from. See WEBSOCKETS.MD §4.1
Fan-out / multicast	Native PUB/SUB available (unused today)	Application-level fan-out only
In-instance node routing	JeroMQMultiplexRouter + JeroMQDemultiplexRouter + JeroMQCommandServer. Hand-rolled state machine dispatching ZMQ-identity frames to per-node backend sockets, plus openRouteToNode / closeRoutesViaInstance control protocol to keep it in sync	URL paths: /cluster/v1/instance/<id> for master, /cluster/v1/node/<id> for workers. Jetty resolves the path; no application-side routing table. Endpoints registered/unregistered on node start/stop
Per-peer connections	One ROUTER socket multiplexes many TCP conns inside libzmq	One Session per TCP conn in Jetty's session table
In-process latency floor	sub-µs over inproc://	~µs over loopback. Irrelevant for control-plane RPC
Test ergonomics	inproc:// + multiple ZContext; integration tests currently enabled = false (see JEROMQ.md §6.6)	Standard WebSocketClient against in-process Server; the existing Jetty test patterns
Protocol evolution	Coordinated cluster-wide deploy	Path-versioned (/cluster/v1, /rt/v2); blue/green-able
Connection-state surface	Hidden behind libzmq	Explicit Session lifecycle we must manage
Removes the §5 design tension	Partially (Phase 2 reduces it, doesn't eliminate it (still libzmq mutex underneath)	Yes) there is no libzmq
Throws away existing code	No (every phase is in-place refactor	Yes) rt-*-jeromq modules retire at Phase 6

---

4. Effort and critical path

Order-of-magnitude estimates. Both columns assume the same engineer-week throughput.

Phase	Renovate (eng-days)	Replace (eng-days)
0	2	2 (shared)
1	3	3–5
2	10–15 (hardest phase: thread-model rewrite)	5–8
3	3–5	3–5
4	5–10 (upgrade + chase 0.6.0 hangs)	2–3
5	3	5 (coexistence requires CI on both transports)
6	3	3 (deletion)
7	open-ended (strategic question)	3
Total	~30–40 eng-days	~25–35 eng-days

Critical-path observations:

• Renovate path's risk is concentrated in Phase 2 (thread-model rewrite) and Phase 4 (0.6.0 upgrade hang chase). Both could overrun.
• Replace path's risk is concentrated in Phase 1 (does the prototype work end-to-end?) and Phase 2 (cancellation semantics + outbox correctness). Phase 1 is a small, cheap probe. If it fails, we know early.
• Phase 0 is identical and valuable regardless of which path we take.

---

5. Risk profile

Risk	Renovate	Replace
Pinned to a dependency the maintainers can't keep stable	High (JeroMQ has years of open context-teardown bugs (JEROMQ.md §10)	None) Jetty WS is well-maintained
Architecture is unproven	Low for both (the invocation pipeline above the transport (PayloadReader/PayloadWriter, RemoteInvoker, LocalInvocationDispatcher, SimpleRemoteInvokerRegistry) was end-to-end tested and working at an earlier point in this codebase's life. The current enabled = false integration suite is regression, not greenfield. See JEROMQ.md §1 and §6.6	Inherits the same validated upper layers) only the transport itself is new
Thread-dump readable by the on-call	Medium (Phase 2 helps but libzmq frames are still in the trace	Low) every frame is java.* / org.eclipse.jetty.*
Migration breaks production	Low (in-place fixes	Medium) requires careful coexistence + cutover (Phase 5)
Protocol-design mistake we have to live with	None (protocol unchanged	Medium) we're picking a wire format; mitigated by path-versioning from day 1
Hidden semantic the new code doesn't replicate	Low (same transport	Low) DEALER buffering is moot under K8s SRV gating; native PUB/SUB fan-out unused in tree; in-instance routing replaced cleanly by URL paths
Phase overruns	Phase 2 + Phase 4	Phase 1 + Phase 2
Sunk cost if we change our mind later	High (every fix locks us further into JeroMQ	Low) the WS transport is just a Jetty endpoint

---

6. Decision matrix

Question	Renovate	Replace
Removes the JEROMQ.md §5 design tension?	partially	yes
Reduces thread count?	no	yes
Unifies auth / PKI / observability with the rest of the service?	no	yes
Preserves DEALER send-while-disconnected?	yes	no (mitigated by outbox)
Lower total effort?	no	slightly (5–10 eng-days saved)
Lower risk profile?	no	yes
Lower switching cost if we later want to change direction?	no	yes
Less new code to write and test?	yes	no
Fewer "open questions" at start (WEBSOCKETS.MD §7)?	yes	no

---

7. Recommendation

Run both plans' Phase 0 immediately (shared, valuable, ~2 eng-days). Then run WEBSOCKETS.MD Phase 1 (the prototype, ~3–5 eng-days) in parallel with JEROMQ.md Phase 1 (the definite-bug fixes, ~3 eng-days. Those bugs should be fixed regardless of which transport survives, because the JeroMQ stack will keep running through the migration).

At the end of that ~week, we know:

• Whether the WS prototype hits a structural blocker. If it does, continue with JEROMQ.md Phases 2–7.
• Whether the JeroMQ definite-bug fixes are sufficient to re-enable the integration tests. They should be.

If the WS prototype works, commit to the replace path: JEROMQ.md Phase 2 (the expensive one) is cancelled, the renovation reduces to the bug fixes we'd already do, and the JeroMQ stack stays on life support through the coexistence window in WEBSOCKETS.MD Phase 5 before retiring in Phase 6.

If the WS prototype fails, we've spent a week proving the renovation is the right call, with the test-suite safety net and the definite-bug fixes already landed.

The Phase 0 + Phase 1 investment is the same either way, so the decision-cost of picking later is genuinely zero.

---

8. References

• JEROMQ.md: current-state diagnosis + renovation plan.
• WEBSOCKETS.MD: alternative WS transport plan.

[ptwohig]

Cluster Capacity Reporting: From quality to available / total

Companion to JEROMQ.md, WEBSOCKETS.MD, and RT-TRANSPORT-COMPARISON.md.
Scope: redesign how cluster instances report their load / capacity to the routing layer.
Transport-independence: this design applies whether the cluster RPC fabric stays on JeroMQ or migrates to WebSocket. The model lives in rt-server-cluster, not in any transport-specific module.

---

1. The current model and why it isn't earning its keep

InstanceMetadataContext today returns (quality: double, nodeIds: Set<NodeId>). Quality is in [0.0, 1.0]; the SimpleRemoteInvokerRegistry uses it to weight candidates when picking a peer for a given NodeId.

Problems:

• No production implementation. The only getInstanceQuality() impl in the tree is the integration-test mock returning ThreadLocalRandom.current().nextDouble(). There is no real scorer wired up anywhere. Production deploys would behave as if all peers are randomly-but-equally healthy.
• Opaque semantics. "Quality 0.7" has no defined meaning. Two peers reporting 0.7 may have wildly different conditions. A human reading a log line can't say whether 0.7 is healthy or struggling.
• No defined boundary between routable and excluded. The registry weights by quality, but there's no value of quality that means "do not route here." A peer at 0.01 is still a candidate.
• No alignment with the workload unit. Elements hosts game sessions; the natural unit is "sessions hosted vs sessions allowed." An abstract score loses that information.

Since there is no production scorer, this is a green-field redesign, not a refactor of working code.

---

2. Proposed model

Replace the opaque score with concrete resource counts that the peer self-reports:

public class InstanceMetadata {
    private Set<NodeId> nodeIds;
    private int availableResources;
    private int totalResources;
    // ...
}

InstanceMetadataContext exposes:

public interface InstanceMetadataContext {
    Set<NodeId> getNodeIds();
    int getAvailableResources();
    int getTotalResources();
    AsyncOperation getInstanceMetadataAsync(...);
}

The peer self-reports both numbers. The registry sorts candidates by availableResources (see §3.1). A peer with availableResources <= 0 is excluded from candidate selection for new work; existing connections are unaffected.

The unit of "resource" is application-defined. See §3.3.

---

3. Three decisions to make up front

3.1 Sort by absolute available, or by ratio?

Approach	Behaviour	When it's right
Absolute available	Picks the peer with the most raw headroom. A 20,000-capacity pod with 12,000 free beats a 5,000-capacity pod with 4,500 free.	Favours larger pods naturally. Matches the literal "do you have room?" question.
Ratio available / total	Picks the peer with the highest free fraction. The 5,000-capacity pod beats the 20,000-capacity pod in the example above.	Balances utilization across heterogeneous pods regardless of size.

Recommendation: absolute. The allocation question is "where does the next unit of work fit?" Absolute answers it directly. Ratio is the right metric for autoscaling decisions (when should we add a pod?), which is a separate question that can layer on top.

The reported metadata exposes both numbers, so a future "ratio mode" can be added without changing the wire format.

3.2 Optimistic local debit between refreshes

SimpleRemoteInvokerRegistry.refresh() polls each peer every REFRESH_RATE_SECONDS (default 30 s). Without compensation, a burst of 50 allocations in 5 s will all route to the same peer (the one that reported highest available at the last refresh) and over-allocate it.

Fix: local optimistic debit. The registry keeps a per-peer "allocated since last refresh" counter:

effectiveAvailable(peer) = peer.lastReportedAvailable - localDebit[peer]

When the registry picks peer X to take n units, localDebit[X] += n. When peer X's next refresh response arrives, localDebit[X] is cleared (the peer's new report already reflects those allocations, or close to it).

This is invisible to peers. They keep reporting honestly. The registry just smooths over the staleness.

Edge cases:

• Release before next refresh. If a session ends quickly, the local debit overshoots the peer's true current load. Acceptable: the next refresh corrects it, and the overshoot biases toward routing elsewhere, which is the safe direction.
• Allocation failure. If the registry picks peer X but the WS dial or invocation fails, the local debit must be rolled back. Handle this in the registry's allocation path: increment debit on dispatch, decrement on dispatch failure before the call lands at the peer.
• Race between local debit and refresh response. If a refresh response and an allocation happen concurrently, the simplest correct behaviour is: clear local debit on refresh receipt, then accept that the next few allocations might re-build it from zero. Tighter consistency isn't worth the lock contention.

3.3 How resources are counted (and the ResourceClaim SPI)

Each peer needs to know its own consumption. For game sessions, that's the open-session count held by the WS handler. Trivially countable. For application-defined resources, we need a small new SPI so elements can self-report:

public interface ResourceClaim extends AutoCloseable {
    int getUnits();
    @Override
    void close();
}

public interface ResourceBudget {
    /**
     * Claim {@code units} from the local budget. Returns null (or throws,
     * tbd) if the budget is exhausted. The returned claim must be closed
     * to release the units.
     */
    ResourceClaim claim(int units);

    int getAvailable();
    int getTotal();
}

Usage in an element:

try (var claim = resourceBudget.claim(sessionWeight)) {
    runSession(...);
}  // resources released on close

getAvailableResources() returns total - sum(open claims). The InstanceMetadataContext implementation reads from ResourceBudget and reports.

Open questions for this SPI (need answers before Phase 2 of the migration):

• What happens when claim can't be satisfied? Two options: return null, or throw. Throwing is simpler at call sites; returning null lets callers gracefully fall back. Suggest throw with a typed CapacityExhaustedException. Callers that want graceful fallback can catch.
• Are claims partitioned by element / application, or shared cluster-wide? Suggest a single shared budget per instance for v1. Per-element budgets are a v2 feature (see §4.1).
• How does an element learn total? Either configured (@Named Guice binding, set in ApplicationNodeModuleDefaults) or runtime-derived (e.g., from cgroup limits). Suggest configured. Explicit calibration beats fragile auto-detection.

---

4. Extensions to defer

These are worth flagging so the v1 design doesn't paint us into a corner, but they aren't needed in v1.

4.1 Per-node breakdown

Today the metadata returns one available/total per instance. We could break it down per-NodeId:

Map<NodeId, ResourceAvailability> availabilityByNode;

This would let different nodes on the same instance advertise independent budgets, which is useful if one game (NodeId A) has heavier sessions than another (NodeId B) on the same pod. The v1 design lumps them into a single instance-level pool; per-node split is additive and doesn't break v1 callers.

4.2 Soft vs hard reservations

claim() is binary in v1. Succeeds or fails. A more nuanced model:

• Soft cap: above this, claims succeed but the peer reports itself "over soft-cap" so the registry deprioritizes routing new work here.
• Hard cap: above this, claims fail outright.

Useful for graceful degradation under load. Not needed in v1; would extend the wire format (overSoftCap: boolean) but not break it.

4.3 Capacity-driven autoscaling

Once the cluster reports aggregate available / total across all instances, that aggregate is a natural autoscaling signal. "Total cluster availability < 20% for > 5 min → add a pod." This belongs in the Kubernetes layer, not the cluster RPC layer, but the data feed is what this design provides.

---

5. Surface area of the change

Touched files (modules to update):

• rt-server-cluster/src/main/java/dev/getelements/elements/rt/InstanceMetadata.java: add availableResources, totalResources; remove quality (or deprecate first, see §6).
• rt-server-cluster/src/main/java/dev/getelements/elements/rt/InstanceMetadataContext.java: add getAvailableResources() / getTotalResources(); deprecate getInstanceQuality().
• rt-server-cluster/src/main/java/dev/getelements/elements/rt/remote/SimpleRemoteInvokerRegistry.java: update add(connection, instanceMetadata, builder) to read the new fields; implement the local-debit counter; update candidate-selection to sort by effectiveAvailable and exclude <= 0.
• New SPI in rt-server-cluster (or sdk-spi): ResourceBudget + ResourceClaim. Bound per-instance; injected into elements that allocate.
• rt-server-cluster-node/.../MasterNodeLocalInvocationDispatcher.java and MasterNodeLifecycle.java: refactor to read from ResourceBudget.
• Default ResourceBudget impl: an in-memory counter with a configured total. Configuration key in ApplicationNodeModuleDefaults.
• Test updates: the integration tests currently mock getInstanceQuality() with random doubles. Replace with fixed (available, total) values that exercise both the sort and the exhaustion path.

Untouched:

• Transport layer (whether JeroMQ or WS). The capacity model rides over whatever transport the cluster RPC fabric uses.
• InstanceDiscoveryService. Discovery is "who exists"; capacity is "who has room." Orthogonal.
• Auth / scoping. A peer at 0/10,000 is still authenticated; it just can't take work.

---

6. Migration

This refactor is independent of the JeroMQ→WS transport decision and can ship on either path. Two options for sequencing:

Option A: capacity refactor first, then transport decision. Land §5 against the current JeroMQ codebase. Production gains a real capacity signal immediately. The eventual WS transport inherits the new model without churn.

Option B: capacity refactor bundled with WS transport. Defer until the WS plan reaches WEBSOCKETS.MD Phase 3 (auth + security), where we're already touching the registry path. One coordinated landing.

Suggest Option A. It's the smaller change, the gain is independent of transport, and it removes a known-broken piece (the random-double quality scorer) from production sooner.

Phasing within Option A:

1. Phase 0. Deprecate the old API. Add new fields + methods to InstanceMetadata / InstanceMetadataContext. Keep quality as a derived value (available / total) for backwards compat. Mark getInstanceQuality() @Deprecated. No registry changes yet.
2. Phase 1. Default budget + reporting. Land ResourceBudget SPI with a default in-memory impl. Wire it into the master node and into InstanceMetadataContext. Instances now report real numbers. Registry still sorts by quality (which is now derived). Validate in staging.
3. Phase 2. Registry sort by effectiveAvailable. Update SimpleRemoteInvokerRegistry to sort by available - localDebit, exclude <= 0, implement local debit + rollback on failure. Production cuts over.
4. Phase 3. Element adoption. Elements that allocate (game sessions, etc.) start using ResourceBudget.claim(...). Until they do, only the master-node default counter contributes. Roll element-by-element.
5. Phase 4. Remove quality. After two release cycles, delete getInstanceQuality() and the derived shim.

---

7. Open questions

• What's the default total per instance? Needs calibration. Suggest a configuration key with a deliberately-low default (e.g., 1,000) so operators are forced to set it explicitly. Under-allocation is preferable to silent over-allocation.
• What's the unit? "Sessions" is the obvious answer for the game-server case, but the system should be agnostic. The total is whatever the operator decides; the elements report claims in the same unit. Document this in the SPI Javadoc.
• Behaviour when all peers are at 0. Fail fast with a "no capacity" error, or queue / wait? Suggest fail fast: this is operationally surfaced load shedding, and the alternative is a different shape of unbounded queueing that we already rejected (see WEBSOCKETS.md §4.1).
• Coexistence with multi-tenancy. If two applications share a peer's budget, a bursty tenant can starve a quiet one. Per-tenant or per-element budgets (§4.1) solve this but add complexity. Punt unless a real tenant starves a real customer.

---

8. References

• WEBSOCKETS.MD §4.1: the health-pruning architecture this design extends (SimpleRemoteInvokerRegistry prunes unreachable peers from the routing snapshot; this design adds capacity-based weighting on top).
• JEROMQ.md: current-state diagnosis of the transport layer this design is independent of.
• RT-TRANSPORT-COMPARISON.md: the broader transport decision context.
• rt-server-cluster/src/main/java/dev/getelements/elements/rt/remote/SimpleRemoteInvokerRegistry.java: the registry whose refresh() and candidate-selection logic this design changes.
• rt-server-cluster/src/main/java/dev/getelements/elements/rt/InstanceMetadataContext.java: the SPI being redesigned.