feat: online evals in the SFT entrypoint

[mikasenghaas]

Summary

Online evals for the sft entrypoint, reusing the orchestrator's eval machinery. The handoff is the filesystem, not NCCL: the trainer writes HF weight checkpoints, the inference server reloads them from disk, and a new evaluator process schedules the evals.

• Config: SFTConfig gains [eval] (OnlineEvalConfig — the orchestrator's EvalConfig shape: multiple [[eval.source]] envs with per-source interval / num_examples / group_size / sampling overrides, plus the inference client) and [inference] (the existing InferenceConfig). deployment.num_infer_gpus splits single-node GPUs between inference and the trainer. The eval client is auto-wired from [inference] (including router bypass for admin ops); [ckpt] is auto-enabled since weight checkpoints are the handoff.
• Evaluator: new evaluator entrypoint (EvaluatorConfig, src/prime_rl/orchestrator/evaluator.py). It watches weights/step_{n} for stable HF checkpoints, points the inference server at each eligible one (/update_weights from disk, weight_broadcast.type = "filesystem"), and runs the due evals sequentially per checkpoint so every epoch measures exactly one policy version. Reuses EvalEnvs / EvalSource / EvalSink / EvalRollouts and logs metrics and traces exactly like the RL orchestrator: eval/{env}/{all,effective}/... + eval/{env}/policy_version (= checkpoint step), traces under rollouts/step_{n}/eval/{all,effective}/. The base model is evaluated at step 0 before training (eval.skip_first_step disables), and the final checkpoint always fires every env. It also runs standalone against any weights dir + OpenAI-compatible server.
• Trainer: the SFT loop saves an HF weight checkpoint at every step an eval env is due (in addition to ckpt.interval) — that write is how the SFT loop tells inference a new policy is ready.
• Launcher: with [eval] set, uv run sft composes the full deployment — inference server, one env server per eval source (same deterministic-address contract as rl), evaluator, and the torchrun trainer — with process monitoring and shared-W&B wiring (trainer and evaluator log to one run; new WANDB_SHARED_PRIMARY env var selects the primary process, defaulting to the previous orchestrator behavior).
• Example: examples/basic/reverse-text/sft.toml (and the configs/basic dev mirror) now train with online reverse-text evals on 2 GPUs (1 train + 1 infer).
• Online evals are single-node only for now (the validator rejects multi-node SFT deployments).

Robustness (from a review pass over the PR):

• Evals are auxiliary: a failed weight reload (checkpoint cleaned under the evaluator, misconfigured admin endpoint) skips that step's evals with an error instead of killing the training run; queued examples are drained so they can't leak into a later epoch.
• Stale weights/step_{n} dirs from a previous run are cleaned at launch (all on fresh start, past the resume step on resume) so the evaluator never replays checkpoints from an abandoned run.
• deployment.num_infer_gpus is reconciled with inference.vllm TP/DP (mirrors RLConfig.auto_setup_deployment), so extra inference GPUs become DP ranks instead of sitting idle.
• sft_local installs a SIGTERM handler (it now spawns GPU-holding children beyond torchrun), waits without busy-spinning while the evaluator drains final evals, and only probes GPUs when it actually partitions them — plain SFT keeps its old no-NVML behavior.
• The evaluator (the process that exits last) owns shared-W&B run finalization.
• Pre-existing is_last_step off-by-one fixed: resuming from a checkpoint at step >= max_steps no longer trains forever.

Known follow-up: eval-step weight checkpoints follow the normal ckpt.keep_last/keep_interval retention; consumer-aware retention (delete-after-eval) is left for a future PR, and a config warning + skip-with-warning path covers the interim.

Verification

E2E on 2×RTX PRO 6000 (1 train + 1 infer GPU):

uv run sft @ examples/basic/reverse-text/sft.toml \
  --max-steps 6 --data.batch-size 8 --data.seq-len 1024 \
  --eval.interval 3 --eval.num-examples 4 --wandb

• Evaluator evaluated the base model at step 0, reloaded weights/step_3 and weights/step_6 (final) from disk, and ran the eval epoch after each reload.
• Metrics landed as eval/reverse-text/{all,effective}/... with policy_version 0/3/6, in the same W&B run as the trainer (shared mode, trainer primary); traces under rollouts/step_{0,3,6}/eval/{all,effective}/traces.jsonl with run: {type: "eval", step: N}.
• Truncation dropped 100% → 75% by step 6 (training effect visible through the online evals).
• All processes exited cleanly (launcher exit 0), and a trainer failure mid-run correctly tore down inference, env server, and evaluator with a non-zero exit.
• uv run rl @ examples/basic/reverse-text/rl.toml --dry-run still resolves after the shared get_physical_gpu_ids move.

🤖 Generated with Claude Code