Summary
StuntDouble today is a deterministic response oracle: given a tool call with certain arguments, it returns a predefined output. That's exactly right for unit-testing agent logic, and it's why the library works so well in CI.
But most real agents don't just read — they mutate. They create invoices, update customers, close tickets. Evaluating those agents over multiple turns requires something stateless mocks can't provide: a world that remembers what the agent did. This RFC proposes that StuntDouble grow an optional, episode-scoped state layer so the same mock definitions that power unit tests today can also power reliable multi-turn evaluations — without changing what the library fundamentally is.
I'm the PM on this project and I'm opening this as a discussion, not a spec. The capabilities below are directional; I'd like maintainer and community input on feasibility, API shape, and sequencing before anything gets built.
Why now
Multi-turn agent evaluation has a well-established recipe, proven out by benchmarks like [τ-bench / τ²-bench](https://arxiv.org/abs/2406.12045): implement tools as simple read/write functions over a small JSON database, let the agent act on it across a conversation, then grade the task by comparing the final database state to the state a correct sequence of actions would have produced. One objective outcome, no LLM judge. The same pattern shows up at larger scale in simulation environments like [Surge's EnterpriseBench: CoreCraft](https://surgehq.ai/blog/enterprisebench-corecraft) and [Scale's RL Environments](https://scale.com/blog/rl-environments).
The insight worth borrowing isn't about RL — it's about evaluation reliability. When tools share a consistent world, an agent's writes are visible to its subsequent reads, so the eval exercises the same causal structure the agent will face in production. And when the world's final state is deterministic, grading becomes a simple, objective comparison instead of a judgment call.
StuntDouble already owns the hardest integration problem here — transparently substituting mocks for real tools inside a live LangGraph agent, switched per-invocation by data rather than code. What it's missing is the world.
The gap, concretely
Take the tools from the README's own quick-start: get_customer, list_bills, create_invoice. Now imagine a multi-turn eval — the agent creates an invoice, applies a discount, then answers "what's outstanding for this customer?"
Today, create_invoice and list_bills are independent mocks. The created invoice never shows up in the list unless the scenario author hand-writes both outputs to be consistent — and re-writes them for every trajectory variant. Worse, if the agent takes a slightly different (but valid) path than the author anticipated, the canned outputs no longer make sense, and the eval measures the author's guesswork rather than the agent's behavior. Mocks are stateless by design: every call resolves against scenario_metadata fresh, and nothing persists across calls. That design is a feature for unit tests (isolation, concurrency safety) and a hard ceiling for multi-turn evaluation.
Proposed direction
Four capabilities, roughly in order of leverage. All opt-in; existing stateless mocks keep working unchanged.
1. Episode-scoped world state
A scenario can seed a small document store — collections of plain JSON entities (customers, bills, invoices) — that lives for the duration of one agent run and is isolated per episode. This preserves the library's existing guarantee that concurrent runs never interfere: state belongs to the episode, not the process.
2. Mocks that read and write state
Extend the data-driven mock format so a case can declare effects on the world alongside its output. Illustratively:
create_invoice: match on input → insert a document into "invoices" → return its id
list_bills: → return the current contents of "bills" for this customer
Write-then-read consistency comes for free, and the mock stays truthful no matter what order the agent acts in. For CRUD-shaped tool families you define one small handler set per entity type instead of hand-enumerating every trajectory. This stays true to the flagship authoring mode: scenarios remain data in scenario_metadata, not code.
3. Reproducible runs
The goal: running the same scenario twice with the same agent behavior should produce the same world, every time. Today's dynamic placeholders ({{uuid}}, {{now}}, {{random_int}}) intentionally introduce variation, which is fine for unit tests but undermines state comparison. Scenarios should be able to opt into fully reproducible values so that eval results are stable and any difference between two runs reflects the agent, not the harness. How to achieve that (seeding, controllable clocks, etc.) is an implementation detail for engineering to shape.
4. Verifying the outcome, not just the calls
CallRecorder already answers "did the agent call the right tools, with the right arguments, in the right order?" For a state-mutating agent over multiple turns, the more important question is: did the agent leave the world in the right condition? An agent might take an unexpected-but-valid path to the correct outcome, or make all the expected calls and still corrupt the data.
The proposal is to extend verification to the world itself: assert on the final state at the end of a run (exact or subset match), with a readable diff when it fails. A particularly attractive authoring option, borrowed from τ-bench: define the expected state by replaying a reference sequence of tool calls through the same mocks — so writing a multi-turn eval is just initial fixtures plus the "correct" actions, and the framework derives what the world should look like. Being able to reset the world and re-run a scenario cheaply also makes it practical to measure consistency, not just single-run success.
Non-goals
This proposal takes clear inspiration from RL-style simulation environments — the stateful world, the reset semantics, the final-state verification all come from that lineage, and these changes would incidentally make StuntDouble more useful as a building block in those contexts. But the goal is not to make StuntDouble an RL environment. User simulators, task curricula, reward signals, and training loops live a layer above, and other projects own them well. The scope here is narrower and closer to home: make multi-turn agent evals on LangGraph reliable and objective, using the mocking layer teams already have.
A likely follow-on beneficiary is MCP tool mirroring — a mirrored tool schema often implies its entity type and CRUD verb (the static strategy's verb routing already leans this way), which could eventually auto-scaffold stateful mocks from real tool contracts. I'd treat that as future work, not part of this proposal.
Open questions
- API shape. Should effects be declarative (data-only, as sketched), programmatic (functions over state), or both? The library's layered authoring model (low-level factory → builder → data-driven) suggests both, declarative-first.
- Episode identity. What keys an episode — LangGraph's
thread_id, an explicit scenario id, or something else? What are the lifecycle semantics (create, snapshot, reset, discard)?
- Verification semantics. Exact state comparison vs. subset match vs. field-level ignore lists? Strict comparison is the most objective but brittle to intentional variation.
- Backward compatibility. Can this land as a purely additive module with zero impact on existing stateless users? (My assumption is yes, and I'd consider it a requirement.)
- Performance. Long multi-turn runs mean many state reads/writes per episode — is a simple in-memory store sufficient for the target scale?
If maintainers think the direction is sound, a reasonable first milestone would be a minimal world-state module plus effects support in data-driven mocks, validated by rebuilding one small multi-turn eval end-to-end as an example. Happy to break this into scoped follow-up issues once we align on direction.
References: [τ-bench](https://arxiv.org/abs/2406.12045) · [τ²-bench](https://arxiv.org/abs/2506.07982) · [EnterpriseBench: CoreCraft](https://surgehq.ai/blog/enterprisebench-corecraft) · [Scale RL Environments](https://scale.com/blog/rl-environments)
Summary
StuntDouble today is a deterministic response oracle: given a tool call with certain arguments, it returns a predefined output. That's exactly right for unit-testing agent logic, and it's why the library works so well in CI.
But most real agents don't just read — they mutate. They create invoices, update customers, close tickets. Evaluating those agents over multiple turns requires something stateless mocks can't provide: a world that remembers what the agent did. This RFC proposes that StuntDouble grow an optional, episode-scoped state layer so the same mock definitions that power unit tests today can also power reliable multi-turn evaluations — without changing what the library fundamentally is.
I'm the PM on this project and I'm opening this as a discussion, not a spec. The capabilities below are directional; I'd like maintainer and community input on feasibility, API shape, and sequencing before anything gets built.
Why now
Multi-turn agent evaluation has a well-established recipe, proven out by benchmarks like [τ-bench / τ²-bench](https://arxiv.org/abs/2406.12045): implement tools as simple read/write functions over a small JSON database, let the agent act on it across a conversation, then grade the task by comparing the final database state to the state a correct sequence of actions would have produced. One objective outcome, no LLM judge. The same pattern shows up at larger scale in simulation environments like [Surge's EnterpriseBench: CoreCraft](https://surgehq.ai/blog/enterprisebench-corecraft) and [Scale's RL Environments](https://scale.com/blog/rl-environments).
The insight worth borrowing isn't about RL — it's about evaluation reliability. When tools share a consistent world, an agent's writes are visible to its subsequent reads, so the eval exercises the same causal structure the agent will face in production. And when the world's final state is deterministic, grading becomes a simple, objective comparison instead of a judgment call.
StuntDouble already owns the hardest integration problem here — transparently substituting mocks for real tools inside a live LangGraph agent, switched per-invocation by data rather than code. What it's missing is the world.
The gap, concretely
Take the tools from the README's own quick-start:
get_customer,list_bills,create_invoice. Now imagine a multi-turn eval — the agent creates an invoice, applies a discount, then answers "what's outstanding for this customer?"Today,
create_invoiceandlist_billsare independent mocks. The created invoice never shows up in the list unless the scenario author hand-writes both outputs to be consistent — and re-writes them for every trajectory variant. Worse, if the agent takes a slightly different (but valid) path than the author anticipated, the canned outputs no longer make sense, and the eval measures the author's guesswork rather than the agent's behavior. Mocks are stateless by design: every call resolves againstscenario_metadatafresh, and nothing persists across calls. That design is a feature for unit tests (isolation, concurrency safety) and a hard ceiling for multi-turn evaluation.Proposed direction
Four capabilities, roughly in order of leverage. All opt-in; existing stateless mocks keep working unchanged.
1. Episode-scoped world state
A scenario can seed a small document store — collections of plain JSON entities (customers, bills, invoices) — that lives for the duration of one agent run and is isolated per episode. This preserves the library's existing guarantee that concurrent runs never interfere: state belongs to the episode, not the process.
2. Mocks that read and write state
Extend the data-driven mock format so a case can declare effects on the world alongside its
output. Illustratively:Write-then-read consistency comes for free, and the mock stays truthful no matter what order the agent acts in. For CRUD-shaped tool families you define one small handler set per entity type instead of hand-enumerating every trajectory. This stays true to the flagship authoring mode: scenarios remain data in
scenario_metadata, not code.3. Reproducible runs
The goal: running the same scenario twice with the same agent behavior should produce the same world, every time. Today's dynamic placeholders (
{{uuid}},{{now}},{{random_int}}) intentionally introduce variation, which is fine for unit tests but undermines state comparison. Scenarios should be able to opt into fully reproducible values so that eval results are stable and any difference between two runs reflects the agent, not the harness. How to achieve that (seeding, controllable clocks, etc.) is an implementation detail for engineering to shape.4. Verifying the outcome, not just the calls
CallRecorderalready answers "did the agent call the right tools, with the right arguments, in the right order?" For a state-mutating agent over multiple turns, the more important question is: did the agent leave the world in the right condition? An agent might take an unexpected-but-valid path to the correct outcome, or make all the expected calls and still corrupt the data.The proposal is to extend verification to the world itself: assert on the final state at the end of a run (exact or subset match), with a readable diff when it fails. A particularly attractive authoring option, borrowed from τ-bench: define the expected state by replaying a reference sequence of tool calls through the same mocks — so writing a multi-turn eval is just initial fixtures plus the "correct" actions, and the framework derives what the world should look like. Being able to reset the world and re-run a scenario cheaply also makes it practical to measure consistency, not just single-run success.
Non-goals
This proposal takes clear inspiration from RL-style simulation environments — the stateful world, the reset semantics, the final-state verification all come from that lineage, and these changes would incidentally make StuntDouble more useful as a building block in those contexts. But the goal is not to make StuntDouble an RL environment. User simulators, task curricula, reward signals, and training loops live a layer above, and other projects own them well. The scope here is narrower and closer to home: make multi-turn agent evals on LangGraph reliable and objective, using the mocking layer teams already have.
A likely follow-on beneficiary is MCP tool mirroring — a mirrored tool schema often implies its entity type and CRUD verb (the static strategy's verb routing already leans this way), which could eventually auto-scaffold stateful mocks from real tool contracts. I'd treat that as future work, not part of this proposal.
Open questions
thread_id, an explicit scenario id, or something else? What are the lifecycle semantics (create, snapshot, reset, discard)?If maintainers think the direction is sound, a reasonable first milestone would be a minimal world-state module plus
effectssupport in data-driven mocks, validated by rebuilding one small multi-turn eval end-to-end as an example. Happy to break this into scoped follow-up issues once we align on direction.References: [τ-bench](https://arxiv.org/abs/2406.12045) · [τ²-bench](https://arxiv.org/abs/2506.07982) · [EnterpriseBench: CoreCraft](https://surgehq.ai/blog/enterprisebench-corecraft) · [Scale RL Environments](https://scale.com/blog/rl-environments)