Skip to content

chore: replace vllm-router fork with upstream release - #3250

Draft
mikasenghaas wants to merge 2 commits into
mainfrom
chore/vllm-router-upstream
Draft

chore: replace vllm-router fork with upstream release#3250
mikasenghaas wants to merge 2 commits into
mainfrom
chore/vllm-router-upstream

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Aug 12, 2026

Copy link
Copy Markdown
Member

Summary

  • Consume vllm-router from PyPI (vllm-project/router, >=0.1.15) instead of the pinned wheel from our PrimeIntellect-ai/router fork (v0.1.26).
  • Drop the [tool.uv.sources] wheel URLs and the exclude-newer-package exemption; the dependency now resolves through the normal registry path.
  • Update docs/inference.md to stop calling the router backend "our fork".
  • Shut down the router cleanly on Ctrl-C: the router now runs in its own session, so a terminal SIGINT only reaches the server — the engine drains first, then inference_local terminates the router. Previously the router died instantly with the foreground group and the watcher thread logged a spurious ERROR Router exited with code 0 - shutting down plus a redundant SIGTERM. The watcher also logs a zero exit as a warning now (an external shutdown racing ours, not a crash).

Upstream 0.1.15 covers everything prime-rl invokes: the consistent_hash policy, --intra-node-data-parallel-size, --vllm-pd-disaggregation, --request-id-headers, --worker-startup-timeout-secs, --prometheus-port, and a typed /inference/v1/generate route. Extra request fields (e.g. return_routed_experts) pass through via a serde-flatten catch-all, and response bytes are forwarded verbatim, so single-replica router replay keeps working.

Breaking

Fork-only router features are gone with this swap:

  • Routed-experts merge for P/D disaggregation: upstream does not merge prefill/decode routed-experts payloads. Router replay (trainer.enable_router_replay) with a disaggregated deployment will not return merged expert traces. Single-replica replay is unaffected.
  • JWT auth and per-run usage metrics: platform-side features of the fork; not used by prime-rl entrypoints. Platform deployments that need them should keep deploying the fork binary independently of prime-rl's dependency.

Verification

Ran the reverse-text RL example end-to-end from this branch on 2 GPUs:

uv run rl @ examples/basic/reverse-text/rl.toml --no-wandb --output-dir outputs/router-smoke
  • Router started from the upstream wheel: vllm_router_rs::server: Starting router on 0.0.0.0:8000 | mode: Regular { worker_urls: ["http://localhost:8100"] } | policy: ConsistentHash.
  • All rollouts flowed through the router's /inference/v1/generate route: 20/20 steps, 128/128 trainable per step, 0.0% errors, reward 0.15 → 0.70.
  • Clean shutdown, final checkpoint written at step_20.

Ctrl-C shutdown, verified with a standalone uv run inference --vllm.model Qwen/Qwen3-0.6B and SIGINT to its foreground process group:

  • Before: router died with the group; watcher logged ERROR Router exited with code 0 - shutting down and re-signalled the process.
  • After: engine drains first, then the router logs Received terminate signal, starting graceful shutdown; no ERROR lines, no leftover processes.

mikasenghaas and others added 2 commits August 12, 2026 17:46
Consume vllm-router from PyPI (vllm-project/router, >=0.1.15)
instead of the pinned wheel from the PrimeIntellect-ai/router fork
(v0.1.26). Upstream now covers everything prime-rl invokes: the
consistent_hash policy, --intra-node-data-parallel-size,
--vllm-pd-disaggregation, and a typed /inference/v1/generate route
that preserves extra request fields.

Fork-only features are dropped: the routed-experts merge for P/D
disaggregation, JWT auth, and per-run usage metrics. Single-replica
router replay still works - the generate route passes
return_routed_experts through and forwards response bytes verbatim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The router subprocess shared the terminal's foreground process group,
so Ctrl-C SIGINTed it directly: it died before the engine drained, and
the watcher thread saw the exit before the finally block marked
shutdown, logging a spurious ERROR and re-signalling the process.

Start the router in its own session so only inference_local owns its
lifecycle: the engine drains first, then the finally block terminates
the router. The watcher now also logs a zero exit as a warning - that
is an external shutdown racing ours (e.g. the rl launcher SIGTERMs the
whole tree, router first), not a crash.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant