Skip to content

Runner restart loses stable ACPX record and turn correlation for resumable sessions #443

Description

@johnlanni

Bug description

After the issue-spec Runner or its ECS host restarts, persisted public sessions can remain present while their stable ACPX record and turn-correlation authority is missing. The Runner then reports the session as interrupted and advertises /resume, but cannot prove that resume will reconnect to the original ACPX record or append to the correct historical stream.

Observed with two real Higress read-only jobs:

  • public session s-231fb53426a6aa269624, job job-7887cea7de144179
  • public session s-d152a9274015a73ccda1, job job-cbf0616f4daf3077

After restart reconciliation both persisted public-session records had:

status: interrupted
acpx_record_id: ""

The shared Runner HOME ACPX registry did not contain a provable public-session → record mapping for either session. Nevertheless, the status writeback instructed users to /resume those public session IDs.

Risk

A resume in this state may:

  • create a new ACPX record while presenting it as a resume;
  • restore the wrong agent/history;
  • lose or misattribute the previous turn;
  • attach a Coordinator Summary to the wrong job/turn;
  • make Runner state, ACPX named-session registry, ACPX stream, and shared-HOME state competing authorities.

Required invariant

One durable authority must reconstruct this chain across process and host restart:

GitHub command/comment
→ public_session_id
→ job_id
→ stable acpx_record_id
→ turn_correlation_id
→ Coordinator Summary/result

For a resumable session:

  • the same public session must resolve to the same ACPX record;
  • the record must exist in the scoped shared HOME registry;
  • agent, repository, runner scope, and stream identity must match;
  • the resumed turn must append to that record;
  • no duplicate record may be silently created;
  • the result summary must bind to the resumed job and turn.

Fail-closed /resume

If the public-session record has no stable ACPX record ID, the record is missing, the registry/stream is unreadable, or scope/agent identity does not match, /resume must be rejected with an actionable message to start a fresh /new. It must not silently create a replacement record.

The generated restart-interrupted status comment must not advertise /resume unless resumability has been proven.

Deterministic regression

Add an integration test using real Runner process termination/restart, not only a mocked helper:

/new dispatch
→ ACPX record creation
→ complete or interrupt a turn
→ terminate Runner process
→ restart using the same state and shared HOME
→ /resume the public session
→ assert identical acpx_record_id
→ assert new turn appends to the same stream
→ assert job/turn/summary correlation
→ assert no second record was created

Negative cases:

  • public session exists but acpx_record_id is empty;
  • persisted record ID is absent from the ACPX registry;
  • record belongs to another agent or repository/runner scope;
  • stream/record metadata is unreadable or corrupt;
  • restart occurred during an unfinished turn.

All negative cases must reject /resume and preserve enough diagnostics for a safe fresh /new.

Environment

Observed on the Runner binary built from revision:

fb3ae0329150648aab7f1b29f24007c717e42b23

The Runner uses a repository/profile/runner-scoped shared HOME and Qoder via ACPX.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions