Design: Explicit lifecycle management for runner storage

[johnlanni]

Design: Explicit Lifecycle Management for Runner Storage

• Proposal Issue: 412

Overview

Extend current main's workspace-only cleanup into a focused storage reconciliation subsystem for filesystem resources the runner already creates outside managed Git workspaces:

• stable public-session runtimes under <workspace-root>/.sessions/<runtime-hash>;
• session-scoped PROCESS workspace pool roots under <workspace-root>/.process-workspaces/<pool-hash>;
• existing managed Git workspaces under <workspace-root>/ws-*, still deleted by workspace.Manager.

The first delivery intentionally excludes future ephemeral job scratch, shared build caches, total managed-byte quotas, and periodic full-tree accounting. It solves the present leak, coordinates with the now-implemented state compaction model, and adds a live minimum-free-space admission guard.

The controlled upgrade path must immediately reclaim legacy runtime directories that are provably no longer resumable. Unmatched directories receive persisted orphan observation. Existing retained public sessions must continue /resume with the same acpx/agent identity.

Related artifacts:

• Proposal: Proposal: Add explicit lifecycle management for runner storage #412
• Companion Proposal: Proposal: Separate persistent session state from ephemeral job HOME #413
• Confirmed SPECs: SPEC-412001 through SPEC-412007
• Latest source baseline: origin/main at facd6556efdc960b3e82e9ced8b0dfabf6372e8d

Current Implementation Locations

• internal/commentrunner/jobs/dispatcher.go
  • stableSessionRuntimeRoot still hashes repo + publicSessionID + Abs/Clean(workspacePath) into .sessions/<hash>.
  • stableSessionRuntimePaths places HOME, GH, XDG, Codex, and ACPX runtime data below that root.
  • prepareSessionProcessWorkspaceRoot now creates .process-workspaces/<hash> from repo, session ID, and canonical workspace path.
  • resumeExecutionBinding preserves legacy CWD spelling only when it names the same session clone.
• internal/commentrunner/jobs/reconcile.go
  • cleanupExpiredWorkspaces still sends only WorkspaceMetadata.Path to workspace.Manager.Cleanup.
  • cleanupWorkspacesFromState still uses a first-seen workspace merge and protects active jobs/sessions by workspace ID.
  • async busy reconciliation still calls workspace-only cleanup.
• internal/workspace/workspace.go
  • owns workspace creation, lock, dirty/uncertain checks, retention calculation, path validation, and RemoveAll(workspace.Path).
  • does not own .sessions or .process-workspaces sibling roots.
• internal/commentrunner/state/retention.go
  • current main automatically tombstones terminal jobs and later prunes them by TTL/count.
  • terminal job tombstones discard job.Workspace and job.Acpx.
  • terminal PublicSession records are the /resume anchors, are never tombstoned, and use SessionTTL=30d and MaxTerminalSessions=200 before pruning.
• internal/commentrunner/state/models.go
  • state schema is version 6.
  • PublicSession retains Acpx, Workspace, queue, lock, and session activity timestamps.
• internal/commands/runner.go
  • poll, startup reconcile, one-shot reconcile, workspace cleanup, and async-busy workspace cleanup remain separate entry points.
• internal/processworkspace
  • owns individual PROCESS worktree leases, integration, reconcile, and cleanup; storage reconciliation must not bypass those semantics.

Impact Scope

• New independent storage sidecar and canonical root owner lock.
• Exact recording and lifecycle reconciliation of .sessions runtimes.
• Inventory/retirement of .process-workspaces session pool roots after process-workspace lifecycle reconciliation.
• Coordination with state v6 job tombstoning and public-session pruning.
• Startup, periodic, async-busy, and explicit operator reconciliation.
• Live statfs minimum-free-space admission.
• Controlled migration and real legacy /resume compatibility verification.

Not in scope:

• .jobs or ephemeral HOME production;
• shared Go/npm/Cargo/pip caches;
• storage_max_bytes or scheduled recursive accounting;
• stage-aware sandbox permission profiles;
• managed-network redesign.

Candidate Plans

Plan A: Delete a sibling runtime when its workspace is deleted

Smallest fix, but it cannot preserve the current 30-day public-session /resume lifetime after the 7-day workspace lifetime, cannot handle multiple legacy hashes, and cannot safely classify unmatched directories. Rejected.

Plan B: Sweep sibling directories by age

Unsafe because mtime is not ownership or activity authority and another session/version may still use the path. Rejected.

Plan C: Store all physical resources in RunnerState

Rejected because current state already compacts/prunes at different lifetimes and old binaries rewrite unknown fields. It would also duplicate workspace lifecycle authority.

Plan D: Independent sidecar joined with current state and filesystem

Selected. Current state remains authoritative for sessions, jobs, locks, queues, workspace activity, and compaction. The sidecar records physical identity, prior managed ownership, orphan observation, and deletion progress only.

Decisions

D1. Sidecar is physical inventory, not lifecycle authority

Store <workspace-root>/.storage/state.json:

type StorageState struct {
    SchemaVersion int                         `json:"schema_version"`
    RootIdentity  string                      `json:"root_identity"`
    Resources     map[string]PhysicalResource `json:"resources"`
    UpdatedAt     time.Time                   `json:"updated_at,omitempty"`
}

type PhysicalResource struct {
    ID                string       `json:"id"`
    Kind              ResourceKind `json:"kind"`
    Path              string       `json:"path"`
    Repo              string       `json:"repo,omitempty"`
    WorkspaceID       string       `json:"workspace_id,omitempty"`
    PublicSessionID   string       `json:"public_session_id,omitempty"`
    PhysicalHash      string       `json:"physical_hash,omitempty"`
    FirstObservedAt   time.Time    `json:"first_observed_at,omitempty"`
    CleanupState      CleanupState `json:"cleanup_state,omitempty"`
    CleanupAttemptID  string       `json:"cleanup_attempt_id,omitempty"`
    LastCleanupError  string       `json:"last_cleanup_error,omitempty"`
}

No LastUsedAt, CleanupAfter, byte-accounting cache, or job lifecycle is duplicated. RootIdentity = sha256(canonicalPath(workspaceRoot)). Root mismatch or a newer unsupported sidecar schema permits report-only inventory only.

D2. Resource kinds match current main

First delivery supports:

session_runtime:<repo>:<public-session-id>:<runtime-hash>
session_process_pool:<repo>:<public-session-id>:<pool-hash>

One public session may own multiple historical physical hashes. .jobs is not registered or scanned until Proposal #413 introduces a producer.

A PROCESS pool is not deleted by raw RemoveAll while it contains active or owned worktrees. In the first delivery, storage reconciliation inventories every pool and automatically removes only a proven-retired empty pool when the owning clone exists and processworkspace inspection proves there is no active/non-cleaned lease, ownership marker, registered worktree, or filesystem worktree. Non-empty, dirty, clone-missing, owner-token/registry-incomplete, or otherwise uncertain pools are preserved and reported with operator remediation. Force-abandon and clone-missing non-empty pool recovery are explicit follow-up work.

When a PROCESS worktree causes workspace cleanup to return kept: linked_worktrees, the runner emits a private operator diagnostic and maintenance remediation; this kept reason must not remain silent merely because it is neither failed nor rejected.

D3. PublicSession is a conservative protection anchor, not sufficient resumability proof

Current main retains terminal PublicSession records for 30 days/count, but ResolveResume also requires the session workspace path to exist. Therefore PublicSession presence is a protection input, not by itself proof that /resume can succeed.

An exactly mapped runtime is protected when any of the following holds:

• the session is non-terminal;
• lstat(session.Workspace.Path) shows an existing validated workspace directory;
• a queued/dispatched/running/interrupted job references the public session through job.PublicSessionID or job.DispatchIntent.PublicSessionID;
• the session lock owner or pending queue is non-empty;
• a non-terminal cancellation, restart reconciliation, or equivalent active control-plane action targets the session;
• state, path identity, or activity cannot be established authoritatively.

An exactly mapped runtime is retired_known when the owning retained session is terminal, every active reference above is absent, and lstat(session.Workspace.Path) returns IsNotExist. Any other filesystem error protects the resource. Under current main this session is deterministically not resumable, so runtime deletion does not change its /resume outcome. If future /resume gains missing-workspace reconstruction, this rule must be removed before enabling that behavior.

When a public session is absent, a runtime is retired_known only if prior sidecar or raw pre-normalization state evidence proves exact ownership. Otherwise it is orphan_observed.

D4. Exact legacy mapping is closed and one-to-many

Normal current-state mapping uses only current session.Workspace.Path, passed to stableSessionRuntimeRoot(repo, publicSessionID, workspacePath) and compared to the complete directory hash. No partial hash, mtime ownership, workspace-ID guess, or similarity match is allowed.

The original legacy session.Acpx.CWD spelling is usable only from a raw state backup/snapshot captured before Normalize; normal LoadFile output is not original-path evidence because current main overwrites CWD with Workspace.Path when both are present. Raw CWD evidence must still reproduce the complete current hash and pass lstat/identity validation. It is used only to prove ownership for an owning session absent from current state; it never broadens protection of an otherwise mapped current session.

Because terminal job tombstones drop job.Workspace and job.Acpx, job copies are optional short-lived evidence and are not required for migration correctness.

For .process-workspaces, mapping uses the current canonical prepareSessionProcessWorkspaceRoot identity inputs. Historical unmatched pools use orphan observation and process-workspace safety checks.

D5. State compaction and storage reconciliation are coordinated but separate

State save remains free of filesystem deletion. Storage reconciliation consumes current state after compaction.

Classification:

• retained PublicSession + exact physical mapping + any D3 protection condition → managed/protected;
• retained terminal session + exact mapping + workspace confirmed missing + no active reference → retired_known;
• no retained session, but prior sidecar or raw pre-normalization evidence proves ownership → retired_known;
• no retained session and no prior exact sidecar ownership → orphan_observed;
• invalid/symlink/escaping path → rejected.

A session pruned by SessionTTL or MaxTerminalSessions therefore does not force an additional orphan grace only when sidecar/raw evidence already proves ownership. First upgrade usually has no such proof for already-pruned sessions. Sidecar loss/corruption loses that proof, so rebuilt unmatched directories conservatively restart orphan observation.

A future optimization may emit pruned session identities from compaction, but this implementation does not require coupling state save to storage deletion.

D6. One canonical root has one destructive owner

<workspace-root>/.storage/owner.lock is a process-lifetime flock for poll/serve ownership. Lock order and phase discipline are:

root owner lock -> state flock -> session workspace lock (dispatch only) -> sidecar lock -> resource try-lock

Dispatch follows owner → session workspace → sidecar. Reconciliation follows owner → state → sidecar and never acquires session workspace locks; it derives active protection from freshly loaded state and uses resource try-locks only. Resource lock failure protects the resource. No code acquires state/session locks while holding the sidecar lock. Sidecar-held phases perform no control-plane write. Admission/pressured cleanup occurs before session/workspace locks.

Startup acquires owner lock then obtains state lock fail-fast/bounded. Same-state-path old/new concurrency is blocked by the existing state flock. New binaries with different state paths are blocked by owner lock. Old binaries do not know owner lock, so controlled upgrade must stop the old runner first.

Owner locking covers:

• runRunnerPollAsync startup;
• runner poll synchronous cycles;
• runner serve startup and lifetime;
• ownership preflight;
• runRunnerReconcileWithStore;
• runRunnerWorkspaceCleanupWithStore;
• runRunnerAsyncReconcileWithStore, including dispatchBusy=true;
• every one-shot destructive cleanup entry;
• runner storage reconcile.

D7. Known legacy runtime recovery is immediate where proof exists

Controlled first reconciliation:

1. stop old runner and acquire root ownership;
2. back up state and prior sidecar;
3. load current state v6;
4. map every retained PublicSession runtime and PROCESS pool exactly;
5. revalidate existing sidecar ownership;
6. inventory .sessions and .process-workspaces with lstat/no-follow;
7. protect all retained-session resources;
8. immediately delete retired_known runtimes, including retained terminal sessions whose workspace is confirmed missing and which have no surviving job/lock/queue/cancellation/reconciliation reference;
9. classify unmatched entries as orphan_observed;
10. reconcile workspaces through workspace.Manager and PROCESS pools through processworkspace safety;
11. re-read statfs before new dispatch.

The production-dominant case—workspace already removed while retained terminal PublicSession metadata and runtime remain—is exactly mapped from current session.Workspace.Path, classified retired_known after active-reference checks, and reclaimed during the first applied reconciliation. If the PublicSession has already been pruned, immediate deletion requires prior sidecar or raw backup ownership proof; otherwise it remains orphan_observed.

D8. Deletion is recoverable and path-confined

Sidecar transitions:

managed/orphan_observed/retired_known -> eligible -> deleting -> removed

For retained terminal sessions with missing workspaces, persist managed → retired_known in the sidecar with session key, workspace path, and full hash evidence. Do not add retirement fields to RunnerState or PublicSession.

After acquiring the resource try-lock, reload state and re-evaluate every mapping, workspace-missing, and active-reference condition. Any change aborts deletion and protects the resource. Persist eligible → deleting and fsync before filesystem mutation. Missing paths complete idempotently. Failures remain deleting with bounded retry diagnostics. Runtime deletion precedes workspace deletion when both are eligible; runtime failure defers grouped workspace deletion. Independent resources continue. The PublicSession stays until normal TTL/count pruning because runtime removal does not change the already-failing missing-workspace /resume result.

Path safety reuses/refactors canonicalPath, validatePathUnderRoot, strict segment validation, lstat, and final identity checks. Never delete .locks, .storage, lock files, arbitrary root entries, symlinks, root equality, or unexpected file types.

D9. Runtime recording is fail-closed with recovery

Before a new runtime is exposed to sandbox execution, upsert its exact sidecar identity. First upsert failure blocks new dispatch so unmanaged runtimes do not proliferate.

A touch failure does not kill already running work, but blocks subsequent new/resume dispatch using the affected runtime until startup reconciliation or explicit storage reconcile repairs/rebuilds the sidecar.

D10. Minimum-free-space admission uses statfs only

Add storage_min_free_bytes, default disabled, and storage_orphan_grace, default 7 days and configurable.

Before dispatch, and before acquiring session/workspace locks:

1. read live statfs free bytes;
2. if threshold passes, continue with no recursive scan;
3. if below threshold, run locked safe reconciliation;
4. re-read statfs;
5. if still below threshold, keep queued work delayed using existing poll cadence and throttled status, or reject new intake according to existing command semantics.

No storage_max_bytes, periodic recursive accounting, size cache, or sidecar removed tombstone quota is required in the first delivery. Reconcile may measure targets before deletion only to report reclaimed bytes.

D11. Explicit stopped-root maintenance uses the same engine

Add:

issue-spec runner storage reconcile \
  [--state <state.json>] \
  [--workspace-root <root>] \
  --dry-run|--apply

Defaults resolve through the same runner scope path logic as current commands. It performs no issue polling and uses the common owner lock, state loading, sidecar, classification, path checks, workspace/process-workspace reconciliation, and deletion transaction.

D12. Rollback is explicit

Upgrade creates private atomic backups. Stop new runner before starting old runner. Old binary ignores sidecar and continues legacy workspace behavior. Do not run new destructive reconciliation concurrently with old runner.

Re-upgrade treats sidecar as non-authoritative until current state revalidates mappings. Missing/corrupt sidecar rebuilds report-only and restarts orphan observation. Deleted expired bytes are not rollback-restorable; compatibility means retained public sessions and control-plane state continue working.

D13. Diagnostics remain private and bounded

Private diagnostics include resource ID/kind/action/safe reason/attempt ID and optional measured reclaimed bytes. They exclude runtime contents, credentials, environment values, and sensitive public paths. Public status only reports storage pressure delay/failure and an operator-safe diagnostic handle.

Component Changes

New internal/commentrunner/storage

• sidecar schema/store and atomic persistence;
• root identity and process-lifetime owner lock;
• runtime/PROCESS-pool inventory and exact ownership mapping;
• state-compaction-aware classification;
• orphan observation and recoverable deletion;
• no-follow path validation;
• statfs abstraction and reconcile results.

internal/commentrunner/jobs/dispatcher.go

• upsert/touch exact runtime resource before sandbox use;
• record exact session PROCESS pool resource after pool preparation;
• perform statfs admission before session/workspace lock acquisition;
• recover sidecar through startup/explicit reconcile;
• no job-scratch producer.

internal/commentrunner/jobs/reconcile.go

• replace first-seen workspace merge with a conservative PublicSession-centered view where needed;
• invoke common storage reconciliation in startup, periodic, and async-busy cleanup paths;
• preserve existing workspace manager semantics;
• isolate failures per resource.

internal/processworkspace

• expose a safe inspection/reconcile result sufficient to prove a session pool has no active lease, marker, or worktree before pool-root removal;
• retain ownership of individual PROCESS worktree lifecycle.

internal/commentrunner/state

• no new filesystem side effects;
• document and test storage behavior around terminal job tombstones and PublicSession TTL/count pruning;
• optional helper exposing current retention policy to runner storage classification so both use one configured SessionTTL.

internal/commands/runner.go

• owner-lock all destructive entry points, including async busy cleanup;
• add storage_min_free_bytes and explicit storage reconcile command;
• keep removed byte totals local/private.

Upgrade Sequence

1. Stop old runner.
2. Acquire canonical root owner lock and fail-fast state lock.
3. Back up state v6 and sidecar.
4. Classify retained terminal, workspace-missing, inactive exact matches as retired_known; already-pruned resources require sidecar/raw proof.
5. Dry-run measured bytes by protected, retired-known, orphan-observed, and rejected class.
6. Apply immediate retired_known cleanup.
7. Observe unmatched directories with the configured 7-day default grace instead of deleting them.
8. Safely reconcile workspace lifecycle and only proven-empty PROCESS pools.
9. Re-read statfs.
10. Run a real /resume smoke test for at least one retained session with a valid workspace when available.
11. Start poll loop.

Failure Semantics

• State unavailable: no destructive cleanup or pressured dispatch.
• Owner/state lock unavailable: no destructive cleanup; focused stop-old-runner diagnostic.
• Backup failure: no first applied migration deletion.
• Sidecar missing/corrupt/root mismatch/newer schema: report-only rebuild, no unmatched deletion.
• Retained-session workspace lstat error other than IsNotExist: protect the runtime.
• Runtime upsert failure: no new dispatch using unmanaged runtime.
• PROCESS pool non-empty, dirty, clone-missing, or ownership-incomplete: preserve it and emit maintenance remediation; do not force-abandon in the first delivery.
• Path identity change/symlink/escape: reject.
• Statfs failure under configured threshold: fail closed for new dispatch.
• Diagnostic failure never broadens deletion eligibility.

Testing Strategy

Unit

• sidecar schema/root mismatch and atomic recovery;
• one session with multiple runtime hashes;
• exact canonical workspace and legacy ACPX CWD mapping;
• state v6 terminal job tombstone with retained PublicSession;
• PublicSession prune classification using prior sidecar ownership;
• lock order, try-only resource locks, and all runner entry points;
• no-follow path/identity-change rejection;
• statfs admission without recursive scan.

Migration/reconciliation

• current sanitized runner fixtures classify retained, retired-known, and orphan-observed resources;
• workspace already removed while retained PublicSession/runtime remain;
• retained terminal workspace-missing session becomes retired-known only after all active references are absent and deletion-time state reload agrees;
• already-pruned/no-proof runtime enters orphan grace while sidecar/raw-proven runtime is retired-known;
• terminal job tombstoned and job.Workspace absent;
• PublicSession within SessionTTL protects runtime;
• PublicSession pruned by TTL/count makes prior sidecar-owned runtime retired-known;
• sidecar loss restarts orphan observation;
• PROCESS pool with active lease/worktree is protected;
• empty retired PROCESS pool is removed;
• uncertain PROCESS pool is preserved and kept: linked_worktrees produces operator diagnostics;
• stopped-root command and async-busy cleanup use identical decisions;
• runner serve conflicts safely with poll/maintenance ownership;
• deletion crash/retry and per-resource failure isolation.

End-to-end

• old /new → upgrade → retained /resume reuses same acpx record;
• no silent replacement session;
• retained terminal session with missing workspace already fails ResolveResume before runtime removal and is no more resumable afterward;
• first apply reclaims current known-expired bytes where ownership proof exists;
• low-disk cleanup/recheck delays or allows dispatch;
• rollback/re-upgrade revalidates sidecar;
• no normal poll full-tree accounting.

Immediate Recovery Acceptance Fixture

Fixture modeled on current runner roots includes:

• mapped runtime for retained PublicSession with a valid workspace;
• exact retained terminal runtime whose workspace is missing and has no active references;
• runtime whose PublicSession was pruned but prior sidecar/migration snapshot proves ownership;
• runtime whose PublicSession was pruned without historical ownership proof;
• runtime with workspace already legacy-deleted;
• unmatched orphan;
• multiple runtime hashes for one session;
• empty and active .process-workspaces pools;
• terminal job tombstones without workspace/acpx payload.

First apply must:

• preserve every retained-session runtime that is active or has a valid workspace;
• immediately remove exact retained-session/workspace-missing and historically proven retired-known runtimes;
• observe unmatched directories;
• remove only empty retired PROCESS pools;
• report exact action/reclaimed totals;
• be idempotent.

Rollout

1. Ship sidecar, ownership lock, explicit command, and report-only classification.
2. Validate state-v6/current-runner fixtures and real dry-run.
3. Enable immediate retained-session/workspace-missing and historically proven retired-known cleanup in controlled upgrade.
4. Keep unmatched orphan deletion behind persisted grace.
5. Enable optional minimum-free-space admission.
6. Leave byte quotas/accounting and job scratch to follow-up work.

Non-Goals

• No session HOME redesign.
• No job scratch producer.
• No shared build cache.
• No total managed-byte quota or periodic full-tree accounting.
• No force-abandon of non-empty, dirty, clone-missing, or ownership-incomplete PROCESS pools.
• No complete sandbox permission profile.
• No TASK/PROCESS execution, Implement issue, code, commit, push, or PR is authorized merely by this Design revision.

Implementation Checklist

• Add sidecar and root-owner lock.
• Record one-to-many runtime and PROCESS-pool physical resources.
• Integrate current state compaction/PublicSession retention semantics.
• Implement retained/retired-known/orphan-observed classification.
• Implement workspace-missing current-session retirement, historically proven retired cleanup, and configurable 7-day-default orphan grace.
• Add PROCESS linked-worktree kept diagnostics and defer uncertain-pool force-abandon.
• Add recoverable no-follow deletion and per-resource isolation.
• Add statfs minimum-free-space admission.
• Add stopped-root reconciliation command and cover all runner cleanup entry points.
• Add state-v6 migration fixtures and real legacy /resume compatibility tests.

SPEC Coverage

• SPEC-412001: D1, D2, D9.
• SPEC-412002: D3, D5, D7, D8.
• SPEC-412003: D2, D5, PROCESS pool/orphan reconciliation.
• SPEC-412004: D6, D10, D11.
• SPEC-412005: D6, D8, D13.
• SPEC-412006: D3, D5, D7, upgrade fixture.
• SPEC-412007: D6, D11, D12.

[johnlanni]

Issue Body Update Summary

• Issue: Design: Explicit lifecycle management for runner storage #414
• Target: Design: Explicit lifecycle management for runner storage #414

Completed current implementation locations, impact scope, candidate-plan comparison, and selected architecture.

[johnlanni]

Issue Body Update Summary

• Issue: Design: Explicit lifecycle management for runner storage #414
• Target: Design: Explicit lifecycle management for runner storage #414

Reworked storage design after review: independent sidecar, one-to-many runtime identity, single-root ownership, immediate known-expired upgrade cleanup, persisted orphan observation, rollback semantics, bounded accounting, and #413 scratch staging.

[johnlanni]

Issue Body Update Summary

• Issue: Design: Explicit lifecycle management for runner storage #414
• Target: Design: Explicit lifecycle management for runner storage #414

Finalized reviewer consensus: unified resume retention, exact legacy deadline/path derivation, try-only lock discipline with phase separation, fail-fast startup, root identity/schema guards, sidecar fail-closed recovery, and bounded accounting defaults.

[johnlanni]

Agent: Coordinator
Type: TASK
ID: TASK-414001
Status: ready
Scope: storage sidecar and root-ownership foundation
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment)
• PR: N/A

Task: Build the storage sidecar, root ownership, and path-safety foundation

Introduce the independent physical-resource sidecar and single-root cleanup authority, including the final owner/state/session/sidecar lock discipline agreed against current main.

Implementation Checklist

• Add the versioned .storage/state.json schema and atomic fsync persistence for runtime and PROCESS-pool physical evidence, without adding retirement fields to RunnerState.
• Implement RootIdentity and report-only behavior for root mismatch or newer sidecar schema.
• Implement owner lock coverage for synchronous/asynchronous runner poll, runner serve, startup/full/async-busy cleanup, maintenance reconcile, and every destructive internal entry.
• Enforce dispatch order owner→session-workspace→sidecar and reconcile order owner→state→sidecar→resource-try-lock; prohibit sidecar→state/session reverse acquisition.
• Refactor/reuse canonical path, strict child validation, lstat/no-follow, final identity checks, and immutable .locks/.storage exclusions.
• Add tests for serve/poll/maintenance ownership conflicts, state flock fail-fast behavior, lock order, sidecar crash recovery, root/schema rejection, symlink escape, and path identity changes.

Execution Planning

• Owned modules / write areas:
  • internal/commentrunner/storage
  • internal/workspace path/lock primitives
  • internal/commands runner poll and serve ownership wiring
• Shared touchpoints:
  • state FileStore
  • dispatcher session workspace lock
  • runner startup/reconcile composition
• Dependency / interface assumptions:
  • Current main state schema remains unchanged for retirement
  • No .jobs producer and no PROCESS force-abandon
• Coupling class: high
• Recommended execution mode: serial foundation before reconciliation/admission tasks
• Complexity / split guidance: High-risk shared foundation with explicit deadlock-prevention tests.

Covers

• SPEC-412001
• SPEC-412005
• SPEC-412007

[johnlanni]

Agent: Coordinator
Type: TASK
ID: TASK-414002
Status: ready
Scope: exact runtime retirement and bounded PROCESS-pool reconciliation
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment)
• PR: N/A

Task: Implement exact runtime retirement and bounded PROCESS-pool reconciliation

Classify active/resumable resources, immediately retire exact workspace-missing terminal runtimes, preserve unproven orphans for seven days by default, and keep uncertain PROCESS pools report-only.

Implementation Checklist

• Map current runtimes only from session.Workspace.Path using the complete current hash; use legacy Acpx.CWD only from raw pre-Normalize backup evidence with exact validation.
• Scan jobs by PublicSessionID/DispatchIntent, session locks/queues, cancellations, and reconciliation references to build the full protection set.
• Classify retained terminal exact mappings with lstat-confirmed missing workspaces and no active references as retired-known; classify already-pruned resources as retired-known only with sidecar/raw proof.
• Persist managed→retired-known→eligible→deleting in the sidecar, acquire resource try-lock, reload state, revalidate all conditions, then remove idempotently; never write retirement to RunnerState.
• Persist unmatched paths as orphan-observed with configurable storage_orphan_grace defaulting to seven days; sidecar rebuild restarts grace.
• Inventory PROCESS pools, protect active evidence, remove only proven-empty retired pools, and emit maintenance diagnostics for non-empty, dirty, clone-missing, ownership-incomplete, or linked-worktree-kept cases.
• Keep workspace deletion in workspace.Manager and isolate failures per resource.

Execution Planning

• Owned modules / write areas:
  • dispatcher/reconcile protection view
  • storage classification/deletion
  • processworkspace empty-pool inspection and diagnostics
• Shared touchpoints:
  • state v6 compaction
  • workspace cleanup diagnostics
  • ACPX resume identity
• Dependency / interface assumptions:
  • Depends on TASK-414001
  • Future missing-workspace resume reconstruction must remove the retirement rule first
• Coupling class: high
• Recommended execution mode: serial after storage foundation with focused data-safety review
• Complexity / split guidance: Core lifecycle task; PROCESS force-abandon remains follow-up.

Covers

• SPEC-412001
• SPEC-412002
• SPEC-412003
• SPEC-412005
• SPEC-412006

[johnlanni]

Agent: Coordinator
Type: TASK
ID: TASK-414003
Status: ready
Scope: minimum-free-space admission and maintenance CLI
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment)
• PR: N/A

Task: Add statfs admission and explicit storage reconciliation

Prevent filesystem exhaustion and expose safe upgrade/stopped-root reconciliation with configurable seven-day orphan grace and no total-byte accounting system.

Implementation Checklist

• Add storage_min_free_bytes and storage_orphan_grace (default seven days) configuration validation.
• Run statfs admission before session/workspace locks; under pressure reconcile safely, recheck, and delay/reject dispatch when required.
• Implement the stopped-root storage reconcile dry-run/apply command with the common owner lock, classification, and transaction engine.
• Dry-run and apply report measured bytes by protected, retired-known, orphan-observed, and rejected class without fixed GiB promises or sensitive contents.
• Cover synchronous/asynchronous poll, async-busy cleanup, runner serve, and maintenance command ownership paths.
• Test statfs failure fail-closed behavior, no normal-poll recursive accounting, owner conflict, stopped-root cleanup, and grace persistence.

Execution Planning

• Owned modules / write areas:
  • storage statfs/results
  • commentrunner config
  • runner storage CLI and serve/poll integration
• Shared touchpoints:
  • dispatcher admission
  • existing poll cadence
  • private diagnostics
• Dependency / interface assumptions:
  • Depends on TASK-414001 and TASK-414002 common reconciler
  • No storage_max_bytes or periodic full-tree accounting
• Coupling class: medium-high
• Recommended execution mode: serial after reconciler interfaces stabilize
• Complexity / split guidance: Focused operational and command integration task.

Covers

• SPEC-412004
• SPEC-412005
• SPEC-412007

[johnlanni]

Agent: Coordinator
Type: TASK
ID: TASK-414004
Status: ready
Scope: first-upgrade recovery and resume verification
Links:

• Proposal Issue: N/A
• Design Issue: N/A
• Implement Issue: N/A
• Related Comments: Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment), Proposal: Add explicit lifecycle management for runner storage #412 (comment)
• PR: N/A

Task: Prove first-upgrade recovery and resume compatibility

Create release-gating state-v6 fixtures and end-to-end evidence for exact workspace-missing retirement, already-pruned orphan handling, lock safety, and bounded PROCESS-pool behavior.

Implementation Checklist

• Fixture retained sessions with valid workspaces, retained terminal sessions with missing workspaces, already-pruned runtimes with and without sidecar/raw proof, unmatched paths, and terminal job tombstones.
• Verify exact retained-session/workspace-missing runtimes are reclaimed on first apply while active or valid-workspace sessions keep the same ACPX record.
• Verify already-pruned/no-proof runtimes enter seven-day-default orphan grace and sidecar/raw-proven runtimes become retired-known.
• Verify ResolveResume already fails before deletion for retained missing-workspace sessions and is no more resumable afterward.
• Test raw pre-Normalize CWD evidence versus normal LoadFile output, deletion-time state reload races, crash recovery, backup failure, rollback/re-upgrade, and sidecar loss.
• Test empty PROCESS pool deletion, active pool protection, uncertain pool report-only behavior, and linked-worktree kept diagnostics.
• Test runner serve/poll/maintenance owner conflicts and document stop-old-runner, dry-run, apply, statfs recheck, and valid-session resume smoke test.

Execution Planning

• Owned modules / write areas:
  • state-v6 migration fixtures
  • runner/acpx end-to-end tests
  • upgrade documentation
• Shared touchpoints:
  • storage sidecar/reconciler
  • state retention
  • processworkspace
  • runner poll/serve/maintenance
• Dependency / interface assumptions:
  • Depends on TASK-414001 through TASK-414003
  • Tests use sanitized copies and never clean live roots
• Coupling class: medium
• Recommended execution mode: release gate after implementation; fixture preparation may start earlier
• Complexity / split guidance: Broad verification centered on agreed immediate-recovery boundary.

Covers

• SPEC-412002
• SPEC-412003
• SPEC-412004
• SPEC-412005
• SPEC-412006
• SPEC-412007

[johnnotify]

Issue Body Update Summary

• Issue: Design: Explicit lifecycle management for runner storage #414
• Target: Design: Explicit lifecycle management for runner storage #414

Realigned Design with current main facd655: PublicSession TTL/count resume authority, implemented state compaction, current PROCESS pools, no job scratch, and statfs-only minimum-free-space admission.

[johnnotify]

Issue Body Update Summary

• Issue: Design: Explicit lifecycle management for runner storage #414
• Target: Design: Explicit lifecycle management for runner storage #414

Applied final maintainer/Qoder consensus: workspace-missing retained sessions can retire safely, pruned/no-proof runtimes use a 7-day orphan grace, retirement remains sidecar-only, serve/dispatch lock order is explicit, and uncertain PROCESS pools are report-only.