UiPath
diff --git a/‎.github/workflows/pr-checks.yml‎
Lines changed: 11 additions & 8 deletions b/‎.github/workflows/pr-checks.yml‎
Lines changed: 11 additions & 8 deletions
diff --git a/‎docs/AB_EXPERIMENTS.md‎
Lines changed: 1 addition & 0 deletions b/‎docs/AB_EXPERIMENTS.md‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎docs/DIALOG_MODE.md‎
Lines changed: 5 additions & 3 deletions b/‎docs/DIALOG_MODE.md‎
Lines changed: 5 additions & 3 deletions
diff --git a/‎docs/TASK_DEFINITION_GUIDE.md‎
Lines changed: 37 additions & 1 deletion b/‎docs/TASK_DEFINITION_GUIDE.md‎
Lines changed: 37 additions & 1 deletion
diff --git a/‎plugins/coder-eval/reference/criteria.md‎
Lines changed: 1 addition & 1 deletion b/‎plugins/coder-eval/reference/criteria.md‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎pyproject.toml‎
Lines changed: 13 additions & 0 deletions b/‎pyproject.toml‎
Lines changed: 13 additions & 0 deletions
@@ -80,7 +80,7 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
 
       # PHASE 1: Fast checks (fail early)
       - name: Check code formatting (ruff format)
@@ -385,7 +385,7 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
 
       - name: Check code formatting (ruff format)
         run: .venv/Scripts/ruff format --check src/ tests/
@@ -843,8 +843,8 @@ jobs:
           python -m pip install --upgrade "pip>=26.2"  # PYSEC-2026-3721 fix floor
           pip install uv
 
-      - name: Install project dependencies (with codex extra)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex
+      - name: Install project dependencies (with codex + litellm extras)
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
 
       - name: Verify required secrets are present
         run: |
@@ -854,14 +854,17 @@ jobs:
           fi
           echo "CODEX_API_KEY present."
 
-      - name: Run Codex live tests
+      - name: Run Codex + litellm-judge live tests
         run: |
           mkdir -p tmp
           # Run serially: `-n0` overrides the global `-n auto` (addopts).
           # Parallel xdist workers share ~/.codex and race the Codex SQLite
           # state migration (`duplicate column name: thread_id`); serial init
-          # migrates the fresh DB exactly once.
-          .venv/bin/pytest tests/test_codex_agent_live.py \
+          # migrates the fresh DB exactly once. test_litellm_judge_live.py
+          # reuses these same CODEX_* secrets to exercise
+          # checker_context.api_route.route: litellm end-to-end (PR #137
+          # review: "nothing in the repo exercises the feature").
+          .venv/bin/pytest tests/test_codex_agent_live.py tests/test_litellm_judge_live.py \
             -m live -n0 -v --tb=short --strict-markers -ra \
             --junit-xml=tmp/junit-codex-live.xml
 
@@ -878,7 +881,7 @@ jobs:
           passed = total - skipped - errors - failures
           print(f"codex-live passed={passed} skipped={skipped} errors={errors} failures={failures}")
           if passed < 1:
-              sys.exit("test_codex_agent_live.py reported zero PASSED tests (missing API key / silent skip?)")
+              sys.exit("Live Codex/litellm-judge tests reported zero PASSED tests (missing API key / silent skip?)")
           PY
 
       - name: Upload Codex live-test artifacts on failure
 
@@ -130,6 +130,7 @@ From `ExperimentVariant` (`coder_eval/models/experiment.py`):
 | `initial_prompt_file` | str                | Prompt replacement loaded from a file                                                  |
 | `run_limits`          | block              | Per-key cap overrides (`max_turns`, `task_timeout`, token/USD budgets)                 |
 | `driver`              | `tempdir`/`docker` | Sandbox driver — enables tempdir-vs-docker arms                                        |
+| `checker_context`     | dict               | Backend/model override for the evaluation side (judge, simulator) — see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context); **not** currently `-D`-reachable |
 
 The `agent` dict is the lever for most A/B tests. Anything on `AgentConfig` is
 fair game: `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`,
 
@@ -60,9 +60,11 @@ The mechanics:
 - The simulator is a **tools-disabled Claude Code agent** with `allowed_tools: []`, an explicit
   deny-list, and no plugins or settings sources. It is pure text-in / text-out, and it **cannot see
   the sandbox** — no files, no terminal, no agent reasoning. Only what the agent writes in the chat.
-- The simulator shares the coding agent's resolved `ApiRoute`, so backend and model come from the
-  run's routing (`--backend direct` / `--backend bedrock`) rather than from the `simulation:` block.
-  There is no model field here to set.
+- The simulator runs on the run's resolved *evaluation* `ApiRoute` — the coding agent's own route
+  (`--backend direct` / `--backend bedrock`) unless `checker_context.api_route.route` overrides it
+  (see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context)). The **model** is separately
+  pinned by `simulation.model` (see [Simulation](TASK_DEFINITION_GUIDE.md#simulation)), not inherited
+  from the route — so an A/B varying the subject model doesn't silently vary the simulated user too.
 
 **Agent-kind constraint.** The *subject* agent can be any registered kind — the dialog driver only
 calls the agent's `communicate()`, so Codex and plugin agents work. The *simulator*, however, is
 
@@ -35,6 +35,7 @@ Complete reference for defining evaluation tasks in Coder Eval.
   - [llm_judge](#llm_judge)
   - [agent_judge](#agent_judge)
   - [skill_triggered](#skill_triggered)
+- [Checker Context](#checker-context)
 - [Reference Solutions](#reference-solutions)
 - [Pre-Run Commands](#pre-run-commands)
 - [Post-Run Commands](#post-run-commands)
@@ -63,6 +64,7 @@ reference: { ... }                    # Optional reference solution (a directory
 pre_run: [ ... ]                      # Optional pre-run commands (before agent starts)
 post_run: [ ... ]                     # Optional post-run commands
 dataset: { ... }                      # Optional dataset fan-out (one task -> N row-tasks)
+checker_context: { ... }              # Optional: backend/model for the evaluation side (judge, simulator)
 ```
 
 ### `dataset`
@@ -1295,6 +1297,40 @@ Observed label is `"yes"` when either signal is found, else `"no"`. Expected lab
 
 **Typical pattern.** Label each dataset row with its true skill (`expected_skill`, `""` for negatives) and stack one `skill_triggered` criterion per skill against the same dataset — each gets its own confusion matrix from the same agent traces. This is the natural companion to a skill A/B experiment (skill plugin on vs. off); see the [A/B Experiment Guide](AB_EXPERIMENTS.md#recipe-ab-a-skill).
 
+## Checker Context
+
+`checker_context` carries task-authored config for the success-checking side, namespaced by reserved key. Currently the only recognized namespace is **`api_route`**:
+
+```yaml
+success_criteria:
+  - type: llm_judge
+    prompt: "Grade the fix for correctness."
+
+checker_context:
+  api_route:
+    route: bedrock    # which backend the whole eval side (llm_judge/agent_judge/simulator) uses
+    model: claude-haiku-4-5   # model override for that route
+```
+
+- `route` selects which backend the WHOLE evaluation side calls (`llm_judge`, `agent_judge`, and the simulator all share one resolved eval route per run — this isn't per-criterion), **decoupled from the agent's own route** (a Claude agent can be graded by a differently-backed judge, or vice versa). This is a backend *name* (`direct` / `bedrock` / `litellm`), not a route object. For `direct`/`bedrock` credentials are never read from the task, always from the matching environment variables (`ANTHROPIC_API_KEY` for `direct`, `AWS_BEARER_TOKEN_BEDROCK`/`AWS_REGION` for `bedrock`). An unconfigured or unknown backend name raises at dispatch rather than silently falling back. **`route: litellm` dispatches `llm_judge` through the `litellm` library** (the `coder-eval[litellm]` extra, `litellm.acompletion`) rather than assuming one wire protocol — a gateway-routed judge model (e.g. an Azure AI `/openai/v1` deployment) rarely speaks Anthropic Messages, so this lets `model` carry its own provider hint (e.g. `azure/gpt-5.6-luna`) and get that provider's actual request/response shape handled by the library. Unlike the other two backends, `route: litellm` has NO implicit env-var fallback — see `params`/`env_params` below, which is how it's configured. `model` is required for `route: litellm` (there is no default open-weight/gateway model).
+- `model` overrides the model that resolved route uses for **`llm_judge` only** — when the criterion itself leaves `model:` unset (precedence: an explicit per-criterion `model:` always wins; below that, `checker_context.api_route.model`; below that, the built-in `DEFAULT_JUDGE_MODEL`). This floor is deliberate and never the agent's own model — an unpinned judge must grade identically regardless of which model the agent under test is using, so `resolve_evaluation_route` never lets the agent's env-configured model (e.g. `BEDROCK_MODEL`) leak into `route.model` on its own; `route.model` is set only when this override was actually given. This works because every `ApiRoute` (`DirectRoute`/`BedrockRoute`/`LiteLLMRoute`) carries its own `model` field; the orchestrator bakes the override into the resolved route's `model` before any criterion runs, so `llm_judge` just reads `context.route.model` — it never reads `checker_context` directly. **`agent_judge` and the simulator do not honor this override** — `agent_judge`'s sub-agent model comes from the criterion's own `agent:` block (defaulted to a fixed judge model), and the simulator's model is pinned by `SimulationConfig.model` (see [Simulation](#simulation) below) — both independent of `checker_context.api_route.model` by design, for the same "measuring instrument stays fixed" reason.
+- `params`/`env_params` (**`route: litellm` only**) are how the call is actually configured — there is no fallback to the agent's own `LITELLM_BASE_URL`/`LITELLM_AUTH_TOKEN` env vars, since a gateway-routed judge model rarely reuses the agent's own LiteLLM proxy/credential. They also cover any of the dozens of other provider-specific kwargs `litellm.acompletion` accepts (`aws_access_key_id`, `vertex_project`, `api_version`, ...), which have no dedicated field on `LiteLLMRoute`:
+  ```yaml
+  checker_context:
+    api_route:
+      route: litellm
+      model: azure/gpt-5.6-luna
+      params:                       # arbitrary literal passthrough kwargs to litellm.acompletion
+        api_version: "2024-05-01"
+      env_params:                   # param name -> ENV VAR NAME (never the secret itself)
+        api_base: LITELLM_BASE_URL
+        api_key: LITELLM_AUTH_TOKEN
+  ```
+  `params` is merged straight into the `litellm.acompletion(**kwargs)` call — litellm validates the param names itself, so there's no allowlist to keep in sync here. `env_params` maps a kwarg name to the *name* of an environment variable; the value is resolved right before the call, so no secret is ever written into task/experiment YAML — this is how an arbitrary provider's config, including secrets (IAM keys, an Azure AD token, a service-account path, ...), is representable without a dedicated field per provider. `env_params` is resolved after `params`, so it always wins for the same key. Rejected at task-load time if given without `route: litellm`. For an Azure deployment, pin `api_version` via `params` to whatever API version the agent side is actually configured for (e.g. Codex's `CODEX_API_VERSION`) — the judge has no way to inherit it, and a mismatched version can hit a different shape of the same endpoint.
+  **`route: litellm` is `llm_judge`-only** — `agent_judge` and the simulator run as real Claude Code CLI subprocesses that speak the Anthropic Messages protocol, so pointing them at an arbitrary litellm-fronted gateway (which may speak an entirely different wire protocol) isn't representable. The orchestrator rejects the combination at resolution time (a clear error, not a silent misroute) if the task has an enabled `agent_judge` criterion or `simulation.enabled: true` alongside `route: litellm` — use `route: bedrock`/`direct` for those instead.
+
+`checker_context` merges shallow-per-namespace across `default_experiment.defaults.checker_context` → `experiment.defaults.checker_context` → `task.checker_context` → `variant.checker_context` (same 4-layer precedence as `agent`/`simulation`). So a judge-model A/B, or a judge-backend A/B, is a variant-level config change, not an edit to every task YAML.
+
 ## Reference Solutions
 
 A reference solution is always a **directory**, given relative to the task YAML's own directory:
@@ -1577,7 +1613,7 @@ simulation:
 | `check_criteria` | `end_of_dialog` | `end_of_dialog`, `every_turn`, or `both`. |
 | `model` | `anthropic.claude-sonnet-4-6` | Model that plays the simulated user. Auto-translated to the run's backend (Bedrock inference profile / bare Anthropic alias), the same way [`llm_judge`](#llm_judge)'s `model` is. |
 
-The simulator runs as a tools-disabled Claude Code agent sharing the coding agent's `ApiRoute`, so temperature and sampling are resolved at the route level (same `-b` flag as the coding agent) and are not configured on this block. The **model is not**: it is pinned by `model` above. Inheriting it from the route meant `BEDROCK_MODEL` decided who the simulated user was, so an A/B varying the subject model silently varied its interlocutor too. Hold `model` fixed across variants for the same reason you hold a judge model fixed — the simulator is part of the measuring instrument, not the thing being measured.
+The simulator runs as a tools-disabled Claude Code agent on the run's resolved *evaluation* `ApiRoute` (the coding agent's own route unless [`checker_context.api_route.route`](#checker-context) overrides it), so temperature and sampling are resolved at the route level (same `-b` flag as the coding agent by default) and are not configured on this block. The **model is not**: it is pinned by `model` above. Inheriting it from the route meant `BEDROCK_MODEL` decided who the simulated user was, so an A/B varying the subject model silently varied its interlocutor too. Hold `model` fixed across variants for the same reason you hold a judge model fixed — the simulator is part of the measuring instrument, not the thing being measured.
 
 **Semantics:**
 
 
@@ -209,7 +209,7 @@ Optional:
 | `include_tool_calls` | When true, include a summary of the latest agent turn's tool calls (via summarize_commands). No-op when turn_records is unavailable. |
 | `include_dialog` | When true, include the full user<->agent conversation across all turns in the judge prompt. In simulation mode the user side is generated by an LLM simulator and may invent premises — the judge should treat any claim made only by the simulated user as possibly fabricated, and not penalize the agent for going along with it unless the task description contradicts it. |
 | `max_dialog_chars` | Aggregate cap on dialog text rendered into the judge prompt. Prevents an N-turn simulation from blowing out the judge's context window. Per-message truncation uses max_file_chars; trailing turns are dropped when this aggregate budget is exceeded (a degraded note is recorded). |
-| `model` | Judge model id (e.g. 'anthropic.claude-sonnet-4-6'). On a BedrockRoute / DirectRoute the value is auto-translated: trailing '-vN[:M]' suffixes and the 'anthropic.' prefix are stripped where the backend doesn't accept them; on Bedrock the cross-region inference-profile prefix is added based on AWS_REGION. |
+| `model` | Judge model id (e.g. 'anthropic.claude-sonnet-4-6'). Leave unset to fall back to checker_context.api_route.model when set, else the built-in default ('anthropic.claude-sonnet-4-6') — the fallback is never the agent's own model, so an unpinned judge grades identically across agent-model A/Bs. On a BedrockRoute / DirectRoute the value is auto-translated: trailing '-vN[:M]' suffixes and the 'anthropic.' prefix are stripped where the backend doesn't accept them; on Bedrock the cross-region inference-profile prefix is added based on AWS_REGION. |
 | `temperature` | Sampling temperature for the judge model. 0.0 keeps grading deterministic. |
 | `max_tokens` | Output token cap. Defaults to 2000 — large enough for the verbose verdict (score + rationale + a handful of findings) without runaway. |
 | `max_file_chars` | Per-file content truncation applied before building the prompt. |
 
@@ -89,6 +89,19 @@ dev = [
 uipath = [
     "uipath>=2.10.31",
 ]
+# Optional extra that enables the `checker_context.api_route.route: litellm`
+# judge backend (llm_judge only) via the `litellm` library's `acompletion` —
+# it normalizes provider-specific quirks (Azure AI's api_base/api_key shape,
+# max_tokens vs max_completion_tokens naming, unsupported-param drops, ...)
+# so the judge transport doesn't hand-roll per-provider HTTP. NOT the same
+# thing as the LiteLLM PROXY (litellm/start-litellm.sh) the AGENT's own
+# `route: litellm` points at over HTTP — this extra calls the library
+# in-process. Without this extra, the framework still installs and runs;
+# the litellm-route judge path fails at dispatch with a clear hint pointing
+# back here.
+litellm = [
+    "litellm>=1.95.0,<2.0.0",
+]
 # Optional extra that enables Codex agent support:
 #   - CodexAgent implementation using official openai-codex SDK
 # Without this extra, the framework still installs and runs; Codex-dependent