Skip to content

Add an offline ArmLaunch broker contract and simulator #22

Description

@Punisheroot

Context

Issues #6 and #7 already provide the answer-free v4 corpus, PowerPlan, exact
schedule, bounded ArmLaunch, and final-gate identity. The missing broker
contract can be implemented and tested without a native provider. Keeping the
native Codex adapter separate makes this slice assignable without credentials,
budget authority, or sealed material.

Outcome

Add an offline broker that validates one ArmLaunch, verifies a caller-provided
pinned source checkout, invokes an injected arm executor, and records one
immutable bounded ArmExecutionResult. Provide a deterministic simulator for
every arm and failure class.

Scope

  • Define canonical broker request/result identity from manifest, schedule,
    PowerPlan, task, route, split, arm, repetition, pair seed, source, execution
    policy, model policy, and pricing commitments.
  • Accept only ArmLaunch plus bounded public execution policy; forbidden
    evaluator fields must be unrepresentable.
  • Verify the caller-provided checkout SHA and copy/materialize only a disposable
    local snapshot; perform no network clone.
  • Invoke an injected executor with bounded turns, workers, retries, time, output,
    and accounting.
  • Persist one immutable result or explicit failure per launch identity,
    including bounded response bytes/digest, usage, worker/discovery metrics,
    process status, cancellation, and cleanup.
  • Reject duplicate execution, identity drift, malformed input, unsupported arm,
    source mismatch, over-budget request, and adaptive retry.
  • Add a deterministic simulator covering all FinalArm::ALL arms and terminal
    results.

Non-goals

  • Native Codex transport, provider preflight/approval, credentials, paid
    execution, sealed bundle access, oracle evaluation, or focused evaluator argv.
  • Producing FinalObservation, PowerPlan, schedule, BCa report, or a claim.
  • Network source acquisition or retries not present in the frozen schedule.

Acceptance criteria

  • Broker input contains no oracle path/bytes, expected answers, sealed
    location, focused argv, evaluator result, or credential.
  • Source SHA and every launch/policy identity are validated before executor
    invocation.
  • One launch yields exactly one immutable result or explicit failure with
    the same identity; duplicates and rewrites fail closed.
  • Cancellation/cleanup and all accounting fields are bounded and explicit.
  • No result is marked quality-passed or claim-eligible.
  • Simulator tests cover every arm, failure, cancellation, duplicate, source
    drift, budget violation, and restart without a provider.

Starting points

  • crates/needle-bench/src/schedule.rs
  • crates/needle-bench/src/final_gate.rs
  • crates/needle-platform-codex/src/bin/needle-sim-codex.rs
  • docs/BENCHMARKING.md

Validation

Run focused schedule, broker identity, immutable result, source, accounting,
cancellation, restart, and all-arm simulator tests using synthetic v4 fixtures.
No Codex installation, provider credential, or sealed bundle is required.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: benchmarksBenchmark corpus, runners, and statistical evidencearea: runtimeCore runtime and execution lifecyclecodex-firstCodex-first development scopeenhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions