feat: online evals during sft

configs/basic/reverse-text/sft.toml

@@ -1,9 +1,11 @@
-# SFT warmup for reverse-text on 2 GPUs. Dev-sized mirror of examples/basic/reverse-text/sft.toml.
+# SFT warmup for reverse-text on 2 GPUs (1 train + 1 infer for online evals).
+# Dev-sized mirror of examples/basic/reverse-text/sft.toml.
 
 max_steps = 100
 
 [deployment]
-num_gpus = 2
+num_train_gpus = 1
+num_infer_gpus = 1
 
 [ckpt] # Checkpoint at the end of training
 
@@ -17,3 +19,24 @@ batch_size = 32
 
 [optim]
 lr = 2e-5
+
+[eval]
+interval = 25
+num_examples = 32
+
+[eval.sampling]
+max_completion_tokens = 1024
+
+[[eval.source]]
+name = "reverse-text"
+
+[eval.source.env.taskset]
+id = "reverse-text"
+
+[eval.source.env.agent.harness]
+id = "null"
+
+[eval.source.env.agent.runtime]
+type = "subprocess"
+
+[inference]

configs/basic/wordle/sft.toml

@@ -3,7 +3,7 @@
 max_steps = 20
 
 [deployment]
-num_gpus = 2
+num_train_gpus = 2
 
 [ckpt] # Checkpoint at the end of training
 

docs/training.md

@@ -176,6 +176,55 @@ uv run sft @ examples/basic/reverse-text/sft.toml --monitors.wandb
 
 Multi-GPU and multi-node use torchrun under the hood (the `sft` entrypoint manages this for you — see [Scaling § SFT and Torchrun](scaling.md#sft-and-torchrun) for non-default layouts; multi-node SFT goes through [SLURM](scaling.md#slurm)).
 
+### Online Evals
+
+`uv run sft` can evaluate the model on rollout-based envs as it trains, reusing the RL orchestrator's eval machinery. Configure an `[eval]` block — the same shape as `[orchestrator.eval]`: multiple `[[eval.source]]` envs with per-source `interval` / `num_examples` / `group_size` / sampling overrides — plus an `[inference]` block for the vLLM server:
+
+```toml
+[eval]
+interval = 25
+num_examples = 32
+
+[[eval.source]]
+name = "reverse-text"
+
+[eval.source.env.taskset]
+id = "reverse-text"
+
+[eval.source.env.agent.harness]
+id = "null"
+
+[eval.source.env.agent.runtime]
+type = "subprocess"
+
+[inference]
+
+[deployment]
+num_train_gpus = 1  # trainer
+num_infer_gpus = 1  # inference
+```
+
+The launcher starts the inference server, one env server per eval source, and an `evaluator` process next to the trainer. The handoff is the filesystem, not NCCL: the trainer writes an HF weight checkpoint at every step an eval env is due (in addition to `ckpt.interval`), and the evaluator watches `weights/step_{n}`, points the inference server at each stable checkpoint (`/update_weights` reload from disk), and runs the due envs against it — sequentially per checkpoint, so every epoch measures exactly one policy version. The base model is evaluated before the first step (disable with `eval.skip_first_step`), and the final checkpoint always fires every env.
+
+#### Multi-Node (Decoupled Trainer and Inference Pool)
+
+On a `multi_node` deployment (SLURM), the trainer and the eval deployment are **two independent SLURM jobs**. `deployment.num_nodes` sizes the trainer job; `deployment.num_infer_nodes` sizes the eval job, which runs the inference pool (one vLLM engine per DP rank behind a single router, `gpus_per_node / inference.vllm.tensor_parallel_size` engines per node), one env server per eval source, and the evaluator:
+
+```toml
+[deployment]
+type = "multi_node"
+num_train_nodes = 2  # trainer job
+num_infer_nodes = 1  # eval job (inference pool + evaluator)
+
+[inference.vllm]
+tensor_parallel_size = 8
+
+[slurm]
+job_name = "my-run"
+```
+
+The only coupling is weight checkpoints on the shared filesystem, so the jobs' lifetimes are independent: when training finishes, the trainer job exits and releases its nodes even while evals are still running; the eval job keeps draining pending checkpoints and exits after evaluating the final one (`max_steps` — without it the eval job never sees a final checkpoint and holds its allocation until walltime). Trainer and evaluator log to a single shared W&B run across both jobs — the trainer creates it, the evaluator finalizes it. Any train × inference layout works: `num_nodes` and `num_infer_nodes` are fully independent.
+
 ### SFT-Specific Knobs
 
 | Knob | What it controls |
@@ -185,15 +234,17 @@ Multi-GPU and multi-node use torchrun under the hood (the `sft` entrypoint manag
 | `data.seq_len` | Per-sample sequence length |
 | `loss_mask.*` | Which roles contribute to loss (system / user / assistant / tool). |
 | `val.interval` | Run validation every N steps; `val.data` mirrors `data` |
+| `eval.interval` | Run online evals every N steps; see [Online Evals](#online-evals) |
 
 ### Important Metrics
 
 Pulled from the console log and mirrored to W&B.
 
 **Progress and loss:**
 
-- `loss/mean` — main signal. Should decrease through the run.
-- `val/loss` — validation loss when `[val]` is set, logged every `val.interval` steps.
+- `loss/mean`, `loss/perplexity` — main signal. Should decrease through the run.
+- `val/loss`, `val/perplexity` — validation metrics when `[val]` is set, logged every `val.interval` steps.
+- `eval/{env}/...` — online eval metrics when `[eval]` is set, logged at each evaluated checkpoint step.
 - `progress/epoch`, `progress/num_samples`, `progress/num_tokens` — dataset progress.
 - `progress/<subset>/ratio_{samples,tokens}` — when training on multiple HF subsets/splits, the realized mixing ratio.
 
@@ -317,7 +368,7 @@ uv run rl @ rl.toml --no-monitors.file                    # disable the local me
 
 The trainer and orchestrator log into a **single shared W&B run**, so all metrics from both processes land in one place. Shared mode requires the W&B SDK ≥ 0.19.9 and is incompatible with `monitors.wandb.offline = true`.
 
-prime-rl deliberately logs a **large number of metrics** for maximum observability: every rollout metric is emitted per subset (`all`/`effective`), per statistic (`mean`/`max`/`min`/`p10`/`p90`), and per environment alongside a cross-env aggregate, so a multi-env run can emit thousands of series. To keep that navigable, W&B mode **auto-creates an `overview` saved view** on the first run into a project — curating the handful of metrics that matter into `train`, `eval`, `stability`, and `performance` sections (with per-env breakdowns). The view is created once per project and adapts to the run's environments; if a later run uses a different set of environments, a new versioned view (`overview-v2`, …) is created instead of overwriting the first.
+prime-rl deliberately logs a **large number of metrics** for maximum observability: every rollout metric is emitted per subset (`all`/`effective`), per statistic (`mean`/`max`/`min`/`p10`/`p90`), and per environment alongside a cross-env aggregate, so a multi-env run can emit thousands of series. To keep that navigable, every training run (RL and SFT) gets an **auto-created `overview` saved view** curating the handful of metrics that matter into `train`, `eval`, `stability`, and `performance` sections (with per-env breakdowns). The view is created once per project and adapts to the run's environments; if a later run uses a different set of environments, a new versioned view (`overview-v2`, …) is created instead of overwriting the first.
 
 ### Platform Monitoring
 

examples/basic/reverse-text/sft.toml

@@ -8,7 +8,32 @@ name = "PrimeIntellect/Qwen3-0.6B"
 [data]
 name = "willcb/R1-reverse-wikipedia-paragraphs-v1-1000"
 seq_len = 4096
-batch_size = 32 
+batch_size = 32
 
 [optim]
-lr = 2e-5
+lr = 2e-5
+
+[eval]
+interval = 25
+num_examples = 32
+
+[eval.sampling]
+max_completion_tokens = 1024
+
+[[eval.source]]
+name = "reverse-text"
+
+[eval.source.env.taskset]
+id = "reverse-text"
+
+[eval.source.env.agent.harness]
+id = "null"
+
+[eval.source.env.agent.runtime]
+type = "subprocess"
+
+[inference]
+
+[deployment]
+num_train_gpus = 1
+num_infer_gpus = 1

examples/vlm_sft_moe/sft.toml

@@ -8,7 +8,7 @@ optimization_dtype = "bfloat16"
 reduce_dtype = "bfloat16"
 impl = "custom"
 attn = "flash_attention_2"
-ep = 8                      # keep in sync with deployment.num_gpus
+ep = 8                      # keep in sync with deployment.num_train_gpus
 
 [renderer]
 name = "auto"
@@ -52,4 +52,4 @@ save_adapter_separately = true
 
 [deployment]
 type = "single_node"
-num_gpus = 8
+num_train_gpus = 8

packages/prime-rl-configs/pyproject.toml

@@ -6,7 +6,7 @@ readme = "README.md"
 requires-python = "~=3.12.0"
 dependencies = [
     "pydantic>=1.10.13",
-    "prime-pydantic-config>=0.4.2",
+    "prime-pydantic-config>=0.4.3",
     "renderers>=0.1.9",
     "tomli>=2.2.1",
     "tomli-w>=1.2.0",

packages/prime-rl-configs/src/prime_rl/configs/evaluator.py

@@ -0,0 +1,84 @@
+from pathlib import Path
+
+from pydantic import Field, model_validator
+
+from prime_rl.configs.monitors import MonitorsConfig
+from prime_rl.configs.orchestrator import EvalConfig
+from prime_rl.configs.shared import ClientConfig, LogConfig
+from prime_rl.utils.config import BaseConfig
+
+
+class OnlineEvalConfig(EvalConfig):
+    """Online evals against a live inference server, driven by weight checkpoints
+    on disk. Extends the orchestrator ``EvalConfig`` (sources, sampling, intervals)
+    with the client of the inference deployment and evaluator-side knobs."""
+
+    client: ClientConfig = ClientConfig()
+    """Client of the inference server evals run against. Auto-wired from the
+    ``[inference]`` block when the launcher manages the server."""
+
+    env_server_base_port: int = Field(5000, ge=1, le=65535)
+    """First port of the env-server port range: the eval source at position ``i`` is
+    served at ``tcp://127.0.0.1:<base + i>``. Sources with an explicit ``serve.address``
+    keep it instead, without shifting the other sources' ports."""
+
+    max_inflight_episodes: int = Field(128, ge=1)
+    """Maximum eval episodes in flight — one episode is one agent run against an env server."""
+
+    @property
+    def env_addresses(self) -> dict[tuple[str, str], str]:
+        """Where each eval source's env server lives, keyed by ``("eval", resolved_name)``.
+        Same contract as ``OrchestratorConfig.env_addresses``: the launcher binds env
+        servers at exactly these addresses and the evaluator connects to them."""
+        return {
+            ("eval", source.resolved_name): source.serve.address
+            or f"tcp://127.0.0.1:{self.env_server_base_port + index}"
+            for index, source in enumerate(self.source)
+        }
+
+
+class EvaluatorConfig(BaseConfig):
+    """``uv run evaluator``: watch a weights directory for new HF checkpoints, point
+    the inference server at each one (``/update_weights`` from disk), and run the
+    configured evals against the updated weights. The ``sft`` launcher writes this
+    config; it can also be run standalone against any trainer that writes
+    ``weights/step_{n}`` HF checkpoints with ``STABLE`` markers."""
+
+    model: str = "Qwen/Qwen3-0.6B"
+    """Name the inference server serves the model under — the ``model`` field of every
+    eval request and the startup model check. Auto-filled from ``model.name`` by the
+    ``sft`` launcher; the name stays fixed across checkpoint reloads (weights are
+    swapped in place), so per-step results are told apart by ``eval/{env}/policy_version``."""
+
+    eval: OnlineEvalConfig
+    """Eval sources, sampling, intervals, and the inference client."""
+
+    weights_dir: Path | None = None
+    """Directory to watch for ``step_{n}`` HF weight checkpoints. The ``sft`` launcher
+    fills it from ``ckpt.output_dir`` when checkpoints are redirected to another volume;
+    defaults to ``<output_dir>/weights``."""
+
+    output_dir: Path = Path("outputs")
+    """Directory to write outputs to — rollout traces and logs are written as
+    subdirectories. Shared with the trainer."""
+
+    max_steps: int | None = None
+    """Trainer step at which the run ends. The final checkpoint always fires every
+    eval env, and the evaluator exits after processing it. If None, the evaluator
+    runs until terminated."""
+
+    resume_step: int | None = None
+    """Trainer step the run resumed from. When set, the startup (base-model) eval is
+    skipped; set ``eval.retrigger_on_resume`` to re-fire interval-aligned evals at
+    this step."""
+
+    log: LogConfig = LogConfig()
+
+    monitors: MonitorsConfig = MonitorsConfig()
+    """Metric monitors (``monitors.wandb``, ``monitors.file``)."""
+
+    @model_validator(mode="after")
+    def auto_setup_weights_dir(self):
+        if self.weights_dir is None:
+            self.weights_dir = self.output_dir / "weights"
+        return self