You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The checked-in benchmarks/corpus/router-cache directory contains the v4
manifest, campaign, schedule, synthetic PowerPlan and observations, legacy
fixtures, public oracles, and a synthetic sealed bundle. The benchmark index
summarizes their evidence boundary, but the directory has no local guide that
maps each file to its producer, consumer, identity, and eligibility.
This makes it unnecessarily easy for a new contributor to mistake synthetic or
public material for a production holdout or provider-backed observation.
Outcome
Add benchmarks/corpus/router-cache/README.md as a concise fixture map, and link
it from benchmarks/README.md. The guide documents existing files only and does
not change the corpus protocol or any benchmark result.
Scope
Inventory manifest.json, campaign.json, power-plan.json, schedule.json, cost-model.json, minimal-live-pilot.json, the synthetic
calibration/final observation JSONL files, oracles/, synthetic-sealed/,
and the legacy files.
For each group, state its schema/identity role, which current command or
evaluator consumes it (or that it is legacy/unconsumed), and whether it is
answer-free, answer-bearing, evaluator-only, synthetic, legacy, or report
input.
Add a short artifact flow from calibration observations to PowerPlan,
schedule, arm execution, sealed evaluation, FinalObservation, and final
report. Mark which boundaries are implemented, fixture-only, or still lack a
production runner/recorder; do not invent commands.
State prominently that checked-in public/synthetic artifacts are permanently
ineligible as independent provider holdouts or general economic evidence.
Explain that the synthetic sealed bundle exists for evaluator tests and must
never be exposed to an arm runner or provider prompt.
Link the directory guide from benchmarks/README.md.
Adding a provider runner, private holdout material, credentials, paid runs,
new economic claims, or publication authorization.
Moving or renaming benchmark files.
Acceptance criteria
Every current top-level router-cache fixture and both subdirectories are
represented in the guide.
Schema/identity, producer/consumer, and evidence eligibility statements
match current source and docs/BENCHMARKING.md.
Implemented, fixture-only, legacy, and missing production boundaries are
explicit; the guide does not present the open broker/recorder work as
shipped.
The guide clearly separates public answer-free launch material, public
legacy oracles, evaluator-only synthetic sealed material, and observation
inputs.
It states that neither the synthetic PowerPlan nor synthetic final
observations support a provider or savings claim.
No fixture bytes, paths, digests, or behavior change.
The new link from benchmarks/README.md resolves.
Starting points
benchmarks/README.md
docs/BENCHMARKING.md
benchmarks/corpus/router-cache/manifest.json
benchmarks/corpus/router-cache/campaign.json
benchmarks/corpus/router-cache/power-plan.json
benchmarks/corpus/router-cache/schedule.json
crates/needle-bench/src/schedule.rs
crates/needle-bench/src/sealed_oracle.rs
crates/needle-bench/src/final_gate.rs
Validation
Run git diff --check, verify every documented relative path exists, and check
the artifact-flow terminology against the current benchmark schemas and docs/BENCHMARKING.md. No Cargo build, provider credential, sealed private
material, or benchmark execution is required.
Context
The checked-in
benchmarks/corpus/router-cachedirectory contains the v4manifest, campaign, schedule, synthetic PowerPlan and observations, legacy
fixtures, public oracles, and a synthetic sealed bundle. The benchmark index
summarizes their evidence boundary, but the directory has no local guide that
maps each file to its producer, consumer, identity, and eligibility.
This makes it unnecessarily easy for a new contributor to mistake synthetic or
public material for a production holdout or provider-backed observation.
Outcome
Add
benchmarks/corpus/router-cache/README.mdas a concise fixture map, and linkit from
benchmarks/README.md. The guide documents existing files only and doesnot change the corpus protocol or any benchmark result.
Scope
manifest.json,campaign.json,power-plan.json,schedule.json,cost-model.json,minimal-live-pilot.json, the syntheticcalibration/final observation JSONL files,
oracles/,synthetic-sealed/,and the legacy files.
evaluator consumes it (or that it is legacy/unconsumed), and whether it is
answer-free, answer-bearing, evaluator-only, synthetic, legacy, or report
input.
schedule, arm execution, sealed evaluation, FinalObservation, and final
report. Mark which boundaries are implemented, fixture-only, or still lack a
production runner/recorder; do not invent commands.
ineligible as independent provider holdouts or general economic evidence.
never be exposed to an arm runner or provider prompt.
benchmarks/README.md.Non-goals
oracles, evaluator behavior, statistics, or accepted evidence reports.
new economic claims, or publication authorization.
Acceptance criteria
represented in the guide.
match current source and
docs/BENCHMARKING.md.explicit; the guide does not present the open broker/recorder work as
shipped.
legacy oracles, evaluator-only synthetic sealed material, and observation
inputs.
observations support a provider or savings claim.
benchmarks/README.mdresolves.Starting points
benchmarks/README.mddocs/BENCHMARKING.mdbenchmarks/corpus/router-cache/manifest.jsonbenchmarks/corpus/router-cache/campaign.jsonbenchmarks/corpus/router-cache/power-plan.jsonbenchmarks/corpus/router-cache/schedule.jsoncrates/needle-bench/src/schedule.rscrates/needle-bench/src/sealed_oracle.rscrates/needle-bench/src/final_gate.rsValidation
Run
git diff --check, verify every documented relative path exists, and checkthe artifact-flow terminology against the current benchmark schemas and
docs/BENCHMARKING.md. No Cargo build, provider credential, sealed privatematerial, or benchmark execution is required.