Skip to content

v1: make isolated agentic-judge collect generic task artifacts and honor judge image #2195

Description

@hallerite

Context

Post-merge review of #2144 found two coupled gaps in AgenticJudgeEnv(share_runtime=False) now on main:

  1. AgenticJudgeEnv.run() calls agents.solver.run(task) without retaining the solver runtime, then JudgeTask.from_trace() restores solution.state.artifacts. Only HarborTask.finalize() currently populates that field; an ordinary task using TaskData.artifacts or /logs/artifacts/ reaches the fresh judge box with an empty artifact set.
  2. JudgeTask.from_trace() copies solution.task.data.image onto the judge task even in isolated mode. Runtime compilation gives that task image precedence over judge.runtime.image, so a purpose-built judge image is silently replaced by the solver image despite the config contract that the judge runtime is honored when sharing is disabled.

These findings landed on #2152 only because its base branch disappeared after #2144 was squash-merged; they are inherited main behavior, not part of #2152's Harbor diff.

Suggested direction

  • In isolated mode, provision the solver runtime explicitly, run the solver in it, and collect task.data.artifacts (including the convention sweep) after task finalization but before leaving the runtime context.
  • Pass that collected set to the isolated judge task rather than assuming a task subclass populated transient trace state.
  • Do not set the isolated judge task's image from the solver row; let the configured judge runtime image win. Preserve only task-level fields that are intentionally shared.
  • Keep shared mode unchanged.

Done when

  • A plain Task that writes to /logs/artifacts/ is restored into an isolated judge sandbox.
  • Declared artifact paths work without a Harbor-specific finalize() override.
  • An explicit judge.runtime.image is the image used in isolated mode.
  • Shared mode continues to use the live solver workspace and runtime.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions