Skip to content

Document the checked-in router-cache benchmark fixtures #27

Description

@Punisheroot

Context

The checked-in benchmarks/corpus/router-cache directory contains the v4
manifest, campaign, schedule, synthetic PowerPlan and observations, legacy
fixtures, public oracles, and a synthetic sealed bundle. The benchmark index
summarizes their evidence boundary, but the directory has no local guide that
maps each file to its producer, consumer, identity, and eligibility.

This makes it unnecessarily easy for a new contributor to mistake synthetic or
public material for a production holdout or provider-backed observation.

Outcome

Add benchmarks/corpus/router-cache/README.md as a concise fixture map, and link
it from benchmarks/README.md. The guide documents existing files only and does
not change the corpus protocol or any benchmark result.

Scope

  • Inventory manifest.json, campaign.json, power-plan.json,
    schedule.json, cost-model.json, minimal-live-pilot.json, the synthetic
    calibration/final observation JSONL files, oracles/, synthetic-sealed/,
    and the legacy files.
  • For each group, state its schema/identity role, which current command or
    evaluator consumes it (or that it is legacy/unconsumed), and whether it is
    answer-free, answer-bearing, evaluator-only, synthetic, legacy, or report
    input.
  • Add a short artifact flow from calibration observations to PowerPlan,
    schedule, arm execution, sealed evaluation, FinalObservation, and final
    report. Mark which boundaries are implemented, fixture-only, or still lack a
    production runner/recorder; do not invent commands.
  • State prominently that checked-in public/synthetic artifacts are permanently
    ineligible as independent provider holdouts or general economic evidence.
  • Explain that the synthetic sealed bundle exists for evaluator tests and must
    never be exposed to an arm runner or provider prompt.
  • Link the directory guide from benchmarks/README.md.

Non-goals

  • Changing JSON/JSONL content, schema, digests, tasks, schedules, PowerPlan,
    oracles, evaluator behavior, statistics, or accepted evidence reports.
  • Adding a provider runner, private holdout material, credentials, paid runs,
    new economic claims, or publication authorization.
  • Moving or renaming benchmark files.

Acceptance criteria

  • Every current top-level router-cache fixture and both subdirectories are
    represented in the guide.
  • Schema/identity, producer/consumer, and evidence eligibility statements
    match current source and docs/BENCHMARKING.md.
  • Implemented, fixture-only, legacy, and missing production boundaries are
    explicit; the guide does not present the open broker/recorder work as
    shipped.
  • The guide clearly separates public answer-free launch material, public
    legacy oracles, evaluator-only synthetic sealed material, and observation
    inputs.
  • It states that neither the synthetic PowerPlan nor synthetic final
    observations support a provider or savings claim.
  • No fixture bytes, paths, digests, or behavior change.
  • The new link from benchmarks/README.md resolves.

Starting points

  • benchmarks/README.md
  • docs/BENCHMARKING.md
  • benchmarks/corpus/router-cache/manifest.json
  • benchmarks/corpus/router-cache/campaign.json
  • benchmarks/corpus/router-cache/power-plan.json
  • benchmarks/corpus/router-cache/schedule.json
  • crates/needle-bench/src/schedule.rs
  • crates/needle-bench/src/sealed_oracle.rs
  • crates/needle-bench/src/final_gate.rs

Validation

Run git diff --check, verify every documented relative path exists, and check
the artifact-flow terminology against the current benchmark schemas and
docs/BENCHMARKING.md. No Cargo build, provider credential, sealed private
material, or benchmark execution is required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: benchmarksBenchmark corpus, runners, and statistical evidencedocumentationImprovements or additions to documentationgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions