Commit 73efb3b
feat(substrate): opt-in substrate dispatch intake (Phase 5) (#5)
* feat(substrate): opt-in substrate intake — config, credential, claim, enqueue
Adds @genesis-works/substrate-db as a dependency and wires a substrate
dispatch intake path into the runOnce tick. Opt-in per slot: substrate
is disabled by default (config.substrate.enabled=false). Flipping this
flag live changes zero behavior on any slot that has not set it.
Config keys (host config substrate block):
substrate.enabled — bool, default false
substrate.credential — string, credential id (NEVER plaintext password)
substrate.kinds — string[], job kinds this slot handles
substrate.slotId — string, this slot's claim identifier
substrate.leaseSecs — number, default 300 (5 minutes)
substrate.host/port/database/user — Postgres connection params
On each tick (when enabled):
1. Credential resolved via existing encrypted-blob pattern.
2. claimNextJob(slotId, kinds) called against shared Postgres.
Log: "[substrate] claim slot=<id> kinds=<list> result=<jobs.id|none>"
3. Claimed job converted to TaskInput (source="substrate:<jobId>").
4. TaskInput enqueued as a local runtime task for next-tick execution.
Failure conditions (all logged, none crash):
- substrate unreachable → "[substrate] skip reason=db-unreachable error=..."
- credential fail → "[substrate] skip reason=credential-fail error=..."
- claimNextJob null → "[substrate] claim ... result=none" (silent no-op)
Pinned substrate-db SHA: d458200b008efe8be3d37912a71b990d73ff2b17
(arc0btc/substrate-db — private, deployed alongside agent-runtime on slots)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(substrate): lease semantics — epoch-fenced write-back, max_attempts, lease recovery
Adds the epoch-fenced write-back hook and lease recovery cadence to the
substrate intake adapter.
Write-back (after finalizeTaskAttempt):
- If task.source starts with "substrate:", calls runSubstrateWriteBack.
- Reads _substrate_job_id and _substrate_claim_epoch from task payload.
- Calls completeJob(epoch) on success, failJob(epoch) on failure.
- Epoch mismatch (stale executor woke after lease recovery) → logged
no-op: "[substrate] complete-epoch-mismatch ... stale executor, ignored"
- Pre-fencing row (epoch=0, expected>0) → distinct log line: "pre-fencing row, ignored"
- completeJob/failJob return WriteBackResult — conflict is logged, not thrown.
Lease recovery:
- Runs releaseExpiredLeases on the substrate DB when isLeaseRecoveryOwner=true.
- Cadence: leaseRecoveryCadenceSecs (default 60s) using a module-level timer.
- ONE nominated owner runs this — not all slots — to avoid stampede.
- Log: "[substrate] lease-recovery released=<n>"
- Lease expiry measured against Postgres now() — NTP drift irrelevant.
Log lines contract (stable, parseable):
[substrate] claim slot=<id> kinds=<list> result=<jobs.id|none>
[substrate] complete jobs.id=<id> epoch=<n>
[substrate] fail jobs.id=<id> epoch=<n> reason=<...>
[substrate] lease-recovery released=<n>
[substrate] skip reason=credential-fail error=<...>
[substrate] skip reason=db-unreachable error=<...>
[substrate] complete-epoch-mismatch jobs.id=<id> expected=<n> actual=<m> — ...
Tests: 125/125 pass (10 new substrate tests + 115 existing).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(substrate): review-feedback correctness pass — catch widening, init flag, host requirement, write-back guard, idempotency key
Addresses 7 inline review findings (Copilot + secret-mars + arc0btc) on
PR #5 — all substantive items resolved in-tree, no follow-ups left.
**Catch widening (Copilot #4, secret-mars [blocking-risk], arc0btc #1)**
`runSubstrateIntakeTick` previously only caught `claimNextJob`. Both
`jobRowToTaskInput` and `enqueueTask` ran outside the try, so a malformed
JobRow or a local SQLite lock contention escaped as a thrown error from
`runOnce`. Each call site now has its own catch with a distinct skip
reason — `job-parse-fail` and `local-enqueue-fail` — so the contract's
"NEVER throws" guarantee is real. The substrate job stays under lease in
both failure modes; `releaseExpiredLeases` reconciles on the next cycle.
**`substrateDbInitialized` retry on credential fail (Copilot #1, all 3
reviewers seconded)**
The flag was set to `true` BEFORE `createSubstrateConnection` succeeded.
A transient credential read miss at first tick (e.g. encrypted blob not
yet available) permanently disabled substrate intake for the process
lifetime — the contract's documented `[substrate] skip
reason=credential-fail` log line would never re-fire. Flag now sets only
inside the success branch; next tick retries.
**`runSubstrateWriteBack` swallows transient PG blips (arc0btc #3)**
Was the only substrate fn whose Postgres calls were unguarded — a
mid-write-back connection reset would propagate out of `runOnce` even
though the local task already finished cleanly. Wrapped both
`completeJob` and `failJob` in a single try; new
`[substrate] write-back error=<msg> jobs.id=<id>` log line on transient
fail. Lease recovery still reconciles eventually.
**Default host removed (Copilot #3, arc0btc seconded)**
`host` no longer defaults to a hard-coded private IP. When
`substrate.enabled: true`, an explicit `substrate.host` is required —
`createSubstrateConnection` throws a clear error if unset. Closes the
"dev/test slot misconfigured at enabled:true accidentally connects to
prod" footgun.
**Self-contained log fallbacks (secret-mars #5)**
The empty-else branches on `completeJob`/`failJob` `{ ok: false }`
results relied on the substrate-db package emitting its own
`[substrate] complete-epoch-mismatch ...` log. If that package's log
format ever changes, those branches went silent. Added
`[substrate] complete-failed jobs.id=<id> epoch=<n>` and
`[substrate] fail-failed ...` fallbacks so this code is self-contained.
**Idempotency-key threading (arc0btc #2 design, secret-mars idempotency
follow-up)**
Closes the "side-effecting jobs can execute twice when a lease expires
mid-flight" hazard structurally — the fence on `jobs.claim_epoch` only
protects the status write, not the action. `jobRowToTaskInput` now
threads `payload.idempotency_key = "substrate-<job_id>-e<claim_epoch>"`
so downstream side-effecting handlers (email send, PR open, tx
broadcast) can dedup against their own per-handler key store. Bounds the
*consequence* from "action runs twice" to "action runs once, second
attempt no-ops."
Substrate tasks also enqueue with `priority: 1` so they execute on the
same or next tick — the lease window now tracks real execution latency
instead of waiting behind lower-priority work, which shrinks the "lease
expires while task waits in local queue" window further.
**Silent null claim (Copilot #5)**
Contract said null-claim is silent; impl logged `[substrate] claim ...
result=none` every tick. Dropped the log — quiet-tick visibility lives
in successful-claim and idle-dispatch event lines, not substrate
tick-rate noise.
**Other nits (Copilot #2, arc0btc #5)**
- Removed unused `getTaskById` import in `substrate.ts`.
- Documented `substrate` block's shallow-merge behavior in `types.ts`
(unlike `profiles`/`adapters` which deep-merge). Slots that extend a
base and want to flip only `isLeaseRecoveryOwner: true` must repeat
the whole substrate block.
Deferred (already declared out of scope by reviewers):
- `pg_advisory_lock` guard on `isLeaseRecoveryOwner` race (secret-mars
[suggestion]) — multi-slot mis-config protection is a follow-up.
- `_substrate_*` payload-injection vs strict validators (secret-mars
[suggestion]) — naming is defensive prefixing already.
Contract block in src/substrate.ts updated with all new log lines and a
new "Side-effect duplicate-execution guard" section.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(substrate): expand coverage via mock.module — claim/complete/fail/lease-recovery paths
Closes the test-gap finding from arc0btc #4 and secret-mars: prior
suite covered only `resolveSubstrateConfig`, `jobRowToTaskInput`, and
two write-back *early-return* guards via a fakeDb that threw if
touched. Claim, complete, fail, epoch-mismatch no-op, transient
write-back throw, and lease recovery — the "highest correctness
surface" per the PR's own Commit-2 description — were untested.
Rewires the file to use `mock.module("@genesis-works/substrate-db", ...)`
with mock fns per substrate-db export. Local DB is a real
`openDb(:memory:)` so `enqueueTask` exercises the actual sqlite write
path. console.{info,error,warn} are spied per-test for log-line
assertions.
New tests (18, alongside 10 existing for 28 total):
- `runSubstrateIntakeTick`:
- happy claim → enqueues local task, returns `claimed=true` with
correct epoch + jobId, emits `[substrate] claim ...` log
- null claim → silent (no `[substrate] claim` log line; just
`claimed: false`)
- db-unreachable → catches `claimNextJob` throw, logs
`reason=db-unreachable`
- local-enqueue-fail → forces `enqueueTask` to throw by closing the
local db, asserts catch fires + log line includes
`reason=local-enqueue-fail jobs.id=<id>` (proves Copilot #4 /
arc0btc #1 fix is wired)
- `runSubstrateWriteBack`:
- non-substrate source no-op (existing — kept)
- missing `_substrate_job_id` no-op (existing — kept)
- complete happy → calls `completeJob` with `claim_epoch` arg,
`[substrate] complete jobs.id=... epoch=...` log
- complete `{ ok: false }` → emits self-contained fallback
`[substrate] complete-failed` log (proves secret-mars #5 fix)
- complete throws → catches transient PG blip, emits
`[substrate] write-back error=... jobs.id=...` (proves arc0btc #3
fix is wired — write-back no longer propagates out of `runOnce`)
- fail happy → calls `failJob`, logs reason snippet
- fail `{ ok: false }` → emits `[substrate] fail-failed` fallback
- `runSubstrateLeaseRecovery`:
- released > 0 → logs `[substrate] lease-recovery released=<n>`
- released = 0 → no log (quiet success)
- throws → catches, logs `[substrate] lease-recovery error=...`,
does NOT propagate
- `createSubstrateConnection`:
- throws when `substrate.host` is unset (proves Copilot #3 fix —
no implicit default to a hard-coded private IP)
- throws on whitespace-only host
- `jobRowToTaskInput`:
- asserts `payload.idempotency_key = "substrate-<job_id>-e<epoch>"`
(proves idempotency-key threading is wired)
- asserts `priority = 1`
- asserts `_substrate_*` payload fields threaded
- asserts `max_attempts = 1` (substrate handles retry)
- existing source / subject-fallback tests kept
Header comment rewritten to match what's actually covered (Copilot #7).
Unused `mock`/`beforeEach`/`afterEach` imports replaced with the ones
actually used (Copilot #6 — though now they ARE used, so the prior nit
naturally resolves).
Test count: 143 total (28 substrate + 115 existing) — up from 125.
`bunx tsc --noEmit` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Jason Schrader <jason@joinfreehold.com>
Co-authored-by: Claude <noreply@anthropic.com>1 parent ff1951b commit 73efb3b
7 files changed
Lines changed: 1169 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
21 | 24 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
26 | 34 | | |
27 | 35 | | |
28 | 36 | | |
| |||
36 | 44 | | |
37 | 45 | | |
38 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
39 | 56 | | |
40 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
41 | 61 | | |
42 | 62 | | |
43 | 63 | | |
| |||
181 | 201 | | |
182 | 202 | | |
183 | 203 | | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
184 | 252 | | |
185 | 253 | | |
186 | 254 | | |
| |||
519 | 587 | | |
520 | 588 | | |
521 | 589 | | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
522 | 601 | | |
523 | 602 | | |
524 | 603 | | |
| |||
0 commit comments