feat(runner): one runner-scoped shared HOME with per-job scratch - #442
Merged
johnlanni merged 17 commits intoAug 12, 2026
Merged
Conversation
Materializes confirmed SPEC-439001..SPEC-439006 from Proposal higress-group#439 into repository durable specs: shared runner-scoped runtime HOME, per-job scratch, storage lifecycle integration, and legacy home migration. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Phase 1 of the runner-scoped isolated runtime HOME change: storage package foundations only, no dispatcher/sandbox wiring yet. - runtime_home.go: RuntimeScope (hostname, realm, repo, runner), scope hash with the physical-hash shape, .runner-home/<hash> and .job-scratch/<job-id> layouts, and fail-closed 0700 preparation mirroring the dispatcher's private-dir semantics. scope.json pins each home to its scope; mismatches fail closed naming both scopes. - runtime_store.go: the new .storage/runtime.json metadata store with the sidecar Store's exact semantics (flock, atomic writes, missing/ corrupt rebuild with corrupt backup, foreign-root/newer-schema report-only). A separate file keeps v1 binaries safe: they never parse records they cannot classify. - runtime_accounting.go: protected/cache/unknown byte classification for the shared home, the eviction-priority cache dir list, and symlink-skipping measurement for homes and job scratch. - runtime_reconcile.go: owner-locked, idempotent job-scratch reconciliation (active jobs kept, terminal/unknown removed through capability-scoped deletion, foreign names rejected) and cache-only eviction confined to the exact eligible cache dirs. - migrate.go: two-phase legacy per-session home import with content-digest conflict detection (any conflict aborts before any import write), atomic copies with private perms, a monotonic imported/validated/retired ledger, and retirement that routes legacy runtime deletion through the existing engine without fresh orphan grace. - service.go: lazily opened runtime store plus RecordRuntimeHome, RecordJobScratch, CompleteJobScratch, and the migration ledger methods; remove.go gains a reusable capability-scoped tree remover. Refs higress-group#439, design higress-group#440. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…ed HOME" This reverts commit ae58638. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Re-materializes after the clean-cutover correction: SPEC-439005 drops old-binary rollback in favor of the single storage sidecar authority, SPEC-439006 (legacy migration) is superseded, and SPEC-439007 adds the clean-cutover requirement. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…ion machinery Rework the phase-1 storage foundations to the corrected clean-cutover model (maintainer correction on higress-group#439, design higress-group#440): no legacy migration, no resume preservation across upgrade, no migration ledger, no old-binary rollback support. - Delete runtime_store.go (the parallel .storage/runtime.json metadata store) and migrate.go (two-phase legacy import + retirement) with their tests. The existing storage sidecar is the single metadata authority for every physical resource. - storage.go gains runner_home and job_scratch resource kinds recorded as ordinary PhysicalResource entries (scope repo / job repo in the repo segment, empty session segment, scope hash / job ID in the hash segment). - RecordRuntimeHome, RecordJobScratch, and CompleteJobScratch now run locked read-modify-write cycles on the shared sidecar with skip-unchanged upserts; CompleteJobScratch is idempotent and leaves crash consistency to the generic engine sweep. - ReconcileJobScratch and EvictRuntimeCaches iterate sidecar records under the same owner lock, keeping active-job protection, foreign name rejection, dry-run reporting, and second-pass no-op behavior. - AdmitDispatch's pressured pass additionally reclaims stale job scratch and rebuildable home caches (best-effort, bounded diagnostics) once the root carries at least one runner_home record. - RecordSessionProcessPool is the thin pool-only recording variant for the shared layout, where no per-session .sessions/<hash> runtime exists. - New tests cover kind validation, record shape/idempotency, the generic sweep dropping stale scratch records, runner_home records surviving an apply reconcile, and pressured admission reclaiming scratch and caches (gated on shared-layout adoption). Refs higress-group#439, design higress-group#440. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Add JobTmpDir/JobGoTmpDir/JobXDGDataHome/JobXDGStateHome to the sandbox Config. In bubblewrap mode each configured host dir is bound read-write at a fixed path below /tmp/issue-spec-scratch using the same --dir + --bind pattern as the temporary HOME mounts; in explicit unsafe mode the host paths are exported directly. TMPDIR, GOTMPDIR, XDG_DATA_HOME, and XDG_STATE_HOME join the protected environment set so command env cannot override them, and the writable-bind reservation rejects operator binds overlapping the job scratch dirs. Empty fields preserve the legacy behavior exactly. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Wire the dispatcher to the runner-scoped shared runtime layout. A RuntimeIdentity (hostname, profile realm, runner identity) derived from the runner configuration pins every job of a repo to one shared runtime HOME below .runner-home/<scope-hash> and one disposable scratch tree below .job-scratch/<job-id>; the zero identity keeps the legacy per-session layout for existing wiring. Recording stays fail-closed in order: runtime home, session process pool, then job scratch, all before sandbox exposure; terminal completion, failure, cancellation, and restart-recovered terminal transitions remove the job scratch best-effort with a bounded diagnostic on failure. Restart reconcile reuses the shared HOME without preparing new scratch. Shared-home mirror writes go through a single atomic helper: an identical existing file is left untouched (mode enforced) and every real write is temp-file plus rename, so concurrent jobs sharing one runtime home never observe partial gh/codex/profile mirrors and steady-state refreshes do not churn the shared home. The child-profile branch wipes the shared GH config dir only when it is non-empty, since steady-state child dispatches already see it empty and wiping would race concurrent jobs. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…d mirror atomicity Add dispatcher tests for the runner-scoped shared layout: two sessions of one repo receive the identical runtime HOME with distinct per-job scratch, a different repo receives a different home below the same .runner-home root, and the zero runtime identity keeps the legacy .sessions/<hash> layout untouched. Fail-closed recording is pinned in order (home, pool, scratch) with sandbox Prepare never reached on failure, and every terminal transition (success, dispatch failure, queued cancellation, confirmed running cancellation) completes the job scratch exactly once. Mirror tests hammer the shared home with concurrent gh/codex refreshes while a reader validates no torn file is ever observed, and pin the skip-unchanged behavior so steady-state refreshes perform no writes. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…concile Extend runner storage reconcile with a runtime section: when the sidecar tracks runner-scoped shared homes, the report prints one line per home with protected/cache/unknown bytes from MeasureRuntimeHome plus total job-scratch bytes from MeasureJobScratch, byte counts only. A new --evict-caches flag (requiring --apply) runs ReconcileJobScratch and EvictRuntimeCaches after the main pass and prints the reclaimed bytes; --json wraps the main report with the runtime and eviction sections when either is present. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Document the runner-scoped shared HOME (.runner-home/<scope-hash>/) and per-job scratch (.job-scratch/<job-id>/), the storage accounting categories and --evict-caches reclamation, and the explicit breaking cutover: a fresh runner root is required, pre-cutover sessions are not resumable, and the new binary never reads for import, modifies, or deletes an old root. Applied to both runner guides and their zh-CN mirrors. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…ot cutover Add the real-toolchain acceptance (short-skipped, requires go on PATH): two jobs sharing one runtime HOME with distinct per-job scratch reuse the HOME-anchored GOCACHE/GOMODCACHE, an offline second build proves the shared module cache is authoritative, and a concurrent build pair with distinct GOTMPDIRs succeeds with identical outputs. The fast cutover acceptance prepares, reconciles, and evicts against a fresh root while a pre-existing old-style .sessions/<hash>/home fixture in another root is proven byte-identical afterwards. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…roots Extend the shared-layout suite with the dispatch-side cutover acceptance: a shared-layout dispatch against a fresh runner root selects the runner-scoped shared HOME while a pre-existing .sessions/<hash>/home fixture in a separate old-style root stays byte-identical and gains no entries. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Independent review follow-ups for the runner-scoped shared runtime home: - The dispatcher's periodic storage reconcile now also reclaims crash-leftover job scratch on the shared layout, best-effort with a bounded diagnostic, instead of only under disk pressure or through the operator CLI. - ReconcileJobScratch revalidates each job against freshly loaded runner state immediately before deletion (the engine's D8 discipline): a job that turned active or newly known since the pass snapshot aborts its deletion, and a reload failure fails safe. - EvictRuntimeCaches skips homes whose scope has any active job so pressured eviction never breaks in-flight builds, and reports a bounded deferral diagnostic when every home is in use. - runner storage reconcile --json always emits the wrapped schema (report plus optional runtime/eviction sections). - Comment notes: stranded .issue-spec-mirror-* temps are bounded and self-healing, the child-profile gh-dir wipe is a steady-state no-op, and RuntimeIdentityFor inherits profile-resolution stability requirements. - Tests cover the periodic scratch recovery, deletion-time revalidation races, the eviction in-use guard, the stable JSON schema, and intermediate-symlink eviction redirection. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Collaborator
Author
Implementation RationaleIntent. One persistent isolated runtime HOME per runner scope (hostname × profile realm × canonical repo × runner identity) replaces per-session runtime HOMEs, so sessions of one scope share Go/npm caches while exact ACPX/session IDs stay the only routing authority. Each job gets private disposable scratch. Deliberate clean cutover: fresh runner root required; no migration or old-binary rollback (maintainer decision; SPEC-439006 superseded). Exact head: Key decisions (path:symbol/line).
Validation. Post-review finding P1-1 (resolved). Independent review found that the shared HOME made Boundaries / known residual risks.
Review status. Independent read-only review completed on the exact head with zero P0/P1; no outstanding blocking findings. This comment is human review context only and does not certify mergeability. |
The Go module cache materializes directories 0555 and files 0444, and unlink requires a writable parent, so every removal built on removeOpenedTree failed with permission denied for the non-root runner service user: EvictRuntimeCaches could never evict go/pkg/mod, the largest rebuildable cache. Local root runs masked the bug; the non-root CI runner surfaced it through the acceptance test's TempDir cleanup. - removeOpenedTree now relaxes the validated tree owner-writable (directories 0700, files 0600, only where owner write is missing) through the already-opened root capability before unlinking. Every safety check is unchanged: exact confinement, symlink refusal (the walk never follows links; removal unlinks them via the relaxed parent), and SameFile revalidation before the final unlink. The pass is best-effort: an entry that resists relaxation is left for the removal pass, whose error stays authoritative. - RemoveManagedTree keeps its exact semantics for the legacy session/pool deletion paths; fixing the shared helper also fixes the same latent failure when a retired .sessions/<hash> home holds a read-only Go module cache (and process pools holding read-only git objects), now pinned by a regression test. - TestSharedRuntimeHomeReusesGoCachesAcrossJobs registers a cleanup that relaxes the fixture home tree before t.TempDir removal runs (cleanups are LIFO), so a non-root runner no longer fails cleanup. Audit of the other shared-layout tests (77783bd, 5528b5a): they write only 0600/0700 fixtures or use fakes, so no other test lets the go tool write into a TempDir home. - New regressions: eviction of a 0555/0444 module-cache fixture reclaims every byte, RemoveManagedTree deletes a read-only legacy session runtime, and the relaxation pass is pinned euid-independently (modes relaxed, symlinks and out-of-root targets untouched). Verified as the nobody user: without the fix the regressions and the acceptance TempDir cleanup fail with the exact CI permission-denied signature; with it, the whole storage package passes as non-root. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Two review P2s in the shared-home cleanup paths: - makeOpenedTreeWritable trusted the dirent file/dir classification even when a filesystem reports DT_UNKNOWN (some FUSE/NFS/CIFS mounts), so a subdirectory could be chmodded 0600 as a file and strand its children; unknown-type entries are now reclassified through the opened root capability. Directories are also relaxed whenever any owner rwx bit is missing (0o700 predicate, not 0o300): a 0300 directory is writable yet unlistable, so the descent never reached its children. - EvictRuntimeCaches classified homes once at pass start and then removed every cache on that stale snapshot. Each home's first cache removal now revalidates its repo against freshly loaded runner state, mirroring revalidateScratchDeletion: a repo that turned active mid-pass — or a reload failure — skips the home fail-safe with a bounded diagnostic. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
…hared HOME The runner-scoped shared TempHome made per-dispatch whole-file writes to .acpx/config.json race: a dispatch for an agent with no host override removed the whole file, and MaterializeAgentOverride replaced it with a single-agent document, so concurrent dispatches in one runner scope silently lost each other's overrides and acpx fell back to the builtin adapter. Add acpx.ApplyAgentOverride: a per-agent upsert/delete applied under an in-process keyed mutex plus a bounded flock on config.json.lock (~2s retry, then a clear error). The read-modify-write preserves all unrelated top-level fields and peer agents' raw entries, fails closed (no modification) on malformed, oversized, symlinked, or non-regular configs, deletes the file only when it existed solely for overrides, and installs the result with a temp file plus rename so readers never observe partial JSON. MaterializeAgentOverride is reimplemented on top of it so no caller can whole-file clobber, and materializeHostAcpxAgentOverride now prunes only the refreshed agent's entry when the host has no override for it. Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
Add a sequential regression test that drives the production materializeHostAcpxAgentOverride path: materialize a qoder override into the shared runtime HOME, then prune codex with no host override, and assert the qoder entry still parses from the shared HOME. The pre-fix whole-file remove deleted the shared config.json outright, so this fails deterministically there, unlike the concurrent cover. Also note at the keyed-mutex sync.Map that entries persist for the process lifetime (one per distinct lock path; bounded by stable runner homes, with per-call accrual only on the ad-hoc MkdirTemp fallback). Signed-off-by: johnlanni <zty98751@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements Proposal #439 / Design #440: replace per-public-session runtime HOMEs (
.sessions/<hash>/home) with one persistent runner-scoped isolated HOME per(hostname, profile realm, canonical repository, runner identity)at.runner-home/<scope-hash>/, plus per-job disposable scratch (.job-scratch/<job-id>/exported asTMPDIR/GOTMPDIR/XDG_DATA_HOME/XDG_STATE_HOME). Sessions of one runner scope now naturally share Go build/module caches and npm caches instead of duplicating multi-GiB runtime data per session.scope.jsonpinning (SPEC-439003)runner_home/job_scratchsidecar resource kinds under the existing owner lock/reconcile engine; byte accounting (protected/cache/scratch/unknown); cache-only eviction with active-job guard (SPEC-439005)BREAKING OPERATIONAL CUTOVER
This change is a deliberate clean cutover with no in-place migration support (maintainer decision, recorded in #439/#440; SPEC-439006 legacy-migration was superseded):
Planning artifacts
Test plan
go build ./...,go vet ./...cleango test ./... -count=1full suite green (includes new storage/jobs/sandbox/commands tests)TestSharedRuntimeHomeReusesGoCachesAcrossJobs(cross-job GOCACHE/GOMODCACHE reuse + concurrent build pair)Issue-spec managed closing links:
Closes #439
Closes #440
Closes #441