feat: dynamic pass-through vLLM inference metrics to W&B

[mikasenghaas]

Summary

• Replace the hand-curated inference-metric allowlist with a dynamic pass-through: every vllm:* Prometheus family the engines emit is logged to W&B automatically.
• Keys follow inference/{scope}/{metric}/{stat}, with the vllm: prefix stripped. Scopes: agg (all engines), prefill/decode (disaggregated only), and one scope per engine id (server0.0, ...).
• Gauges and counters pass through verbatim; counters add a derived :rate. Histograms log _sum/_count plus :rate/:mean, and scope-level /p50|p90|p99 from pooled bucket deltas. prefix_cache_hit_rate logs /pooled (Δhits/Δqueries over the scope) plus per-engine aggregations (/min, /max, ...).
• Engine identity is {endpoint}.{engine_label}, so DP engines behind one endpoint get separate scopes. One code path covers all deployment types.
• The auto-generated W&B overview gains an inference section: 7 panels pairing the fleet aggregate (mean/sum) with the cross-engine tail that flags a single sick engine (max for pressure, min for health) — KV usage, preemption rate, requests running/waiting, prefix hit rate, generation and prompt token rates.
• Config surface unchanged: collect_inference_metrics toggles collection; inference_metrics_roles is still auto-derived for disaggregated runs.

Breaking

• All inference metric keys are renamed; saved dashboards need updating. Old inference/engine/{id}/vllm:* and the curated names are gone.
• Raw samples are logged each poll instead of a trailing 20-sample mean. Use W&B UI smoothing. This also stops stale metrics from re-logging at their last window average forever.
• running_imbalance/waiting_imbalance are dropped — read max vs mean/min of num_requests_running|waiting. The combined throughput alias is dropped — read prompt|generation_tokens_total:rate/sum.

Verification

E2E on a SLURM H200 cluster (Qwen3-30B-A3B, alphabet-sort, 3 steps per deployment type, W&B project inference-metrics-smoke):

• single-node, dp2 behind one endpoint: engine-label split into server0.0/server0.1 — run afe1a971
• multi-node, 2 replicas × tp4: server0.0…server3.0 — run 6926f621
• disaggregated P/D (NIXL, llm-d router): prefill*/decode* scopes with exact scope-sum consistency (322 prefill prompt tokens = 226 + 96) — run e9098978; the 3-step loop is blocked by pre-existing PD serving issues

Per-run API checks: all five aggregations per metric, per-engine counters sum to the agg sum, rates/quantiles/hit rate non-degenerate under live traffic.

Schema e2e after the key rename: reverse-text RL, 20/20 steps, reward 0.16 → 0.71 — run vllm-metrics-schema-e2e. API-verified: 628 inference/* keys, zero vllm: prefixes, flattened engine scopes, quantiles as /p99, hit rate /pooled + /min, no legacy keys.

---

Note

Medium Risk
Large refactor of orchestrator metrics logging with breaking W&B key renames and behavior change (no smoothing), but limited to observability—not training or inference serving logic.

Overview
Replaces the hand-curated vLLM Prometheus allowlist with dynamic pass-through of every vllm:* family. Scrapes are parsed into per-engine snapshots (EngineSnapshot), logged per engine as inference/{endpoint.engine}/{metric}, and rolled up to inference/agg|prefill|decode/{metric}/{min|max|sum|mean|median} (plus histogram p50/p90/p99 from pooled bucket deltas and :rate / :mean derived from poll-interval deltas).

Breaking observability changes: metric key names change entirely; the 20-sample smoothing window is removed (raw values each poll); curated aliases like throughput, running_imbalance, and fixed-scope counters are dropped in favor of the generic naming scheme.

The W&B overview workspace adds a dedicated inference section with multi-series panels (fleet aggregate vs min/max tail) for KV usage, preemptions, queue depth, prefix cache hit rate, and token rates; legacy inference keys are removed from the performance section.

^{Reviewed by Cursor Bugbot for commit d7b9084. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d7b9084. Configure here.}

[cursor]

Agg generation min hides decode stalls

Medium Severity

The overview pairs fleet generation_tokens_total:rate/sum with /min on the agg scope. In disaggregated runs agg includes prefill engines whose generation rate stays at zero, so the min series is pinned at 0 and cannot flag a stalled decode engine. Decode-scoped min/sum series are logged but this panel does not use them.

 

^{Reviewed by Cursor Bugbot for commit d7b9084. Configure here.}