feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session memory with intent-aware embeddings, FTS ranking, and historian auto-extraction. Fork-lane migrations and an opt‑in distill-skill-memory task were added; notes surface in a <skill-memory> block and project stats now appear in status/TUI.

• New Features

  • Opt-in per-skill recall via SKILL.md (skill-memory: { enabled: true }, inline or block); the skill tool advertises optional intent; before/after hooks stash intent, resolve provenance, and append a <skill-memory> block; ctx_skill_recall exposes on-demand recall without re-loading; ctx_skill_note writes with embed‑on‑write and hash/cosine dedup.
  • Ranking cascade: cosine over intent_embedding + delta_embedding with FTS5 fallback; recall_count tracked; distill-skill-memory (opt‑in) runs a pre‑embed sweep for NULL/stale vectors, then emits a read‑only corpus health report; CLI/schema/dashboard expose the task (default off).
  • Historian auto-extraction: TC: skill(<name>) markers → <skill_observations>; parser threads to runners; both OpenCode and Pi promote to global '*' notes (source_type='historian'), recallable from any project.
  • Status/TUI: adds a “Skill memory” section with per-project totals (notes/skills/pinned); RPC returns skillMemory stats; ctx-status awaits executeStatus; system prompt includes concise ctx_skill_note/ctx_skill_recall guidance.
  • Storage: initializeDatabase creates/repairs skill_memory, the FTS vtable + triggers, and rebuilds FTS when empty; fork-lane migrations v10000–v10002 (table, embeddings/FTS, historian columns + global '*') run via runForkMigrations without advancing the upstream watermark.

• Bug Fixes

  • Recall scope: queries union the skill’s project partition with global '*' so project-local skills surface historian notes; cold‑start tier derivation fixed.
  • Provenance/resolve: accept plain filesystem paths and file://; anchor + last‑match parse; Windows path normalization; name‑based fallback when the provenance line is truncated; project-skill walk stops at the repository worktree root; global discovery handles skill/ and skills/.
  • Stability/guardrails: no crash when the plugin is disabled; session‑keyed intent stash with TTL and per‑session pruning; log recall errors; count per‑note XML framing to stay within max_tokens; clamp pinned budget to min(max_pinned_tokens, max_tokens); reject ctx_skill_note when disabled in frontmatter; TC: skill(<name>) preserves identity (drops only CR/LF/tab).

^{Written for commit 5d9f58c. Summary will update on new commits.}

Greptile Summary

This PR adds skill-memory — per-skill cross-session recall backed by SQLite, with an optional historian auto-extraction pipeline. The implementation is substantial (~7 k lines) and ships with comprehensive tests and a four-phase commit structure that maps cleanly to its stated design phases.

• Transparent recall: three-hook augmentation on the skill tool adds an optional intent param, stashes it pre-validation, and appends a <skill-memory> block to the tool result using a cosine/FTS5/flat recall cascade.
• Storage & migrations: fork-lane migrations (v10000–10002) create skill_memory + FTS5 vtable + historian columns; initializeDatabase now also carries the full schema and a guarded FTS self-heal, so both fresh and upgraded databases converge correctly.
• Historian auto-extraction: TC: skill(<name>) markers cause the historian to emit <skill_observations> blocks; both the OpenCode and Pi runners promote those as global '*' notes; the previous P1 (project-local skills never seeing historian notes) was fixed by unioning the skill's own partition with the global '*' partition in all four recall-path query functions.

Confidence Score: 5/5

• This PR is safe to merge. The four previous blocking issues (cross-session intent eviction, orphaned global notes for project-local skills, FTS self-heal gap, disabled-skill writes) were all resolved in follow-up commits and are verified by regression tests.
• All architectural risks in the initial round of review have been addressed: the intent stash now uses composite sessionID:callID keys, the recall path unions own-partition and global '*' for every query function, the FTS self-heal fires a guarded rebuild on upgrade, and ctx_skill_note rejects notes for disabled skills before any DB write. The remaining findings are style-level: a redundant pure-function call, a footer appended to an explicit tool result, and a missing index for a background dreamer task that is off by default. The codebase is well-tested (dedicated test files for each phase, migration coexistence tests, historian promotion regression tests) and the design is clearly documented.
• No files require special attention. The storage.ts double-call to recallPartitionPredicate in searchSkillMemoryFts and the missing reembed index are worth addressing but do not block merge.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	Core storage layer for skill_memory. Correct partition predicate unioning own + global '*' partition for recall, while keeping write/dedup paths exact-partition. Minor: searchSkillMemoryFts calls recallPartitionPredicate twice (pure but redundant).
packages/plugin/src/features/magic-context/skill-memory/recall.ts	Correct 4-rung recall cascade (embedding cosine → FTS5 fallback → flat recency×hit). Token budget fill handles pinned notes with dual constraints. recallSkillMemoryBlock logs errors instead of swallowing silently (previous review finding addressed).
packages/plugin/src/features/magic-context/fork-migrations.ts	Fork migration runner correctly uses per-row presence checks rather than watermark, preventing version-collision skips. Migrations are idempotent (IF NOT EXISTS + columnExists guards). Ordering is enforced via sort, and each migration runs in its own immediate transaction.
packages/plugin/src/tools/ctx-skill-note/tools.ts	Enabled-guard added (previous P1 addressed). Correct dedup sequence: hash dedup → semantic cosine dedup → insert. Note: passes pre-computed partition key where raw projectIdentity is expected in several storage calls — idempotent due to partitionKey's design but inconsistent with the function signatures.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	Registry-first + disk-fallback resolution works correctly. Correctly passes raw projectIdentity to recallSkillMemoryBlock (correct pattern). The returned block includes the buildSkillMemoryBlock footer reminder ("call ctx_skill_note after using this skill") even on an explicit recall request.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Intent stash uses composite sessionID:callID keys (previous P1 addressed). Before-hook correctly fires pre-schema-validation. After-hook populates skillLoadRegistry and injects recall block in one pass. Non-fatal error handling throughout.
packages/plugin/src/features/magic-context/storage-db.ts	skill_memory and skill_memory_fts tables created in initializeDatabase with full column set. Guarded FTS self-heal (ftsCount===0 && rowCount>0) added (previous P1 addressed). Upgrade path correct: IF NOT EXISTS + fork migrations handle both new and existing DBs.
packages/plugin/src/features/magic-context/skill-memory/reembed.ts	Correctly identifies stale notes (NULL embeddings OR model version mismatch). Uses IS NOT ? for NULL-safe model version comparison. No dedicated index on project_identity alone may cause a full table scan on very large databases, but bounded by LIMIT 200.

Sequence Diagram

sequenceDiagram
    participant Model
    participant Hook as tool.execute hooks
    participant Registry as SkillLoadRegistry
    participant DB as skill_memory (SQLite)
    participant Recall as recallSkillMemoryBlock

    Model->>Hook: tool.execute.before (skill, intent)
    Hook->>Registry: stashIntent(sessionID:callID, intent)

    Model->>Hook: tool.execute.after (skill result)
    Hook->>Hook: parseSkillProvenance(output)
    Hook->>DB: read SKILL.md frontmatter
    Hook->>Registry: set(sessionID:skillId, provenance+config)
    Hook->>Recall: maybeInjectSkillMemory(db, tier, projectIdentity, intent)
    Recall->>DB: embedTextForProject(intent) → vector
    alt Embedding match found
        Recall->>DB: getRankingCandidates → cosine rank (Rung 1)
    else No embedding match
        Recall->>DB: searchSkillMemoryFts → BM25 rank (Rung 3)
    else No intent
        Recall->>DB: getSkillMemoryNotes → recency×hit (Rung 2/4)
    end
    Recall->>DB: bumpRecallCountByIds
    Recall-->>Hook: "<skill-memory> block"
    Hook-->>Model: skill result + appended block

    Note over Model,DB: Write path (ctx_skill_note)
    Model->>DB: ctx_skill_note(skill, kind, delta, intent)
    DB->>DB: hash dedup → cosine dedup → insert
    DB->>DB: FTS5 triggers sync skill_memory_fts

    Note over DB: Historian auto-extraction
    DB->>DB: promoteSkillObservations(observations)
    DB->>DB: "insert tier=global, project_identity='*', source_type='historian'"

Loading

_{Reviews (29): Last reviewed commit: "refactor(migrations): move skill-memory ..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>