research: preserve consolidated method-development evidence - #296
Draft
mattmre wants to merge 39 commits into
Draft
research: preserve consolidated method-development evidence#296mattmre wants to merge 39 commits into
mattmre wants to merge 39 commits into
Conversation
…2-01 Re-runs the SciFact + NFCorpus query-encoder-swap campaigns on post-#276 main (H1 fix: baseline no longer sees the C3a build-time bounded adapter) and regenerates both auto-generated result docs. Also exercises the real all-mpnet-base-v2 swap backend on the cached 3090, closing the two open debt rows. EVIDENCE: experiment_runs/drift-recovery/swap/swap-campaign-manifest-2026-06.json (config.swap_model == all-mpnet-base-v2, record_type == drift_recovery_swap_campaign); regenerated docs/drift-recovery-swap-results-2026-06.md + docs/drift-recovery-swap-nfcorpus-results-2026-06.md. SMOKE (floor-tier): H1-fix regression gate PASSED - C3a baseline == C0/C2/C2O/C4a per seed on both datasets (SciFact seed-42 all 0.793459; NFCorpus seed-42 all 0.555004); scripts/check_block_flag.py -> CLEAR, 0 carried debt rows; test_query_encoder_drift.TestQueryEncoderDriftRealModelSmoke (CHELATED_RUN_REAL_MODEL_TESTS=1) -> Ran 1 test, OK (real backend init logged). CD-H1-01 CLOSED: both docs regenerated; stale-C3a supersession banner dropped (intended end state). Contamination signal was on NFCorpus (old C3a baselines differed, now match); SciFact C3a baseline was already clean pre-re-run. Fresh C3a finals small vs old (SciFact 3-seed ~0.157/0.161/0.164 vs 0.159; NFCorpus 3-seed mean 0.0524 vs 0.0535). C0/C2/C2O finals bit-identical; C4a finals show small re-run variance (SciFact seed-42 0.2359->0.2363; NFCorpus mean 0.0543->0.0529) - run noise, not the H1 baseline signal. CD-A2-01 CLOSED: real swap path run through QueryEncoderDrift._backend in the production campaign driver + opt-in gated smoke passed; default-CI skip remains intentional (17-min HF hang). ## Brutal Honesty - No new L1/L4/L8: no production code changed; docs + evidence artifacts only. - HYGIENE (disclosed): untracked 22 NFCorpus per-run JSONs (25 MB) that PR #275 force-committed; same class as the already-gitignored SciFact per-run files. Only the 14.5 KB summary manifests (which carry the doc-source rows) stay tracked. Removes 25 MB of regenerated blobs from the tracked tree going forward (prior blobs remain in #275 history) and makes NFCorpus consistent with SciFact tracking policy. - L5 (disclosed, pre-existing): the real-model swap smoke is opt-in (CHELATED_RUN_REAL_MODEL_TESTS=1), not default-CI, to avoid a 17-min HF hang; the real path is now covered by the recorded campaign artifact + this manual gated run, not by default CI. - SMOKE tier: floor (regression parity + block-flag + gated smoke), not a new end-to-end fixture; the campaign itself is the end-to-end evidence. BHS_SELF_DRAFT: 100 CARRY_FORWARD: none DEFERRED_SCOPE: H5 post-bank verdict (separate PR); H4 ablation (separate) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lation Runs the merged H5 post-bank head-to-head driver (#290) and the H4 compound_cycles knob (#291) on the query-encoder-swap arena. Rung-13/14 runtime evidence for the lattice program. EVIDENCE: docs/drift-recovery-post-bank-headtohead-results-2026-06.md (SciFact), docs/drift-recovery-post-bank-headtohead-nfcorpus-results-2026-06.md (NFCorpus), docs/drift-recovery-h4-compound-cycles-ablation-2026-07.md; summary manifests + H4 run JSONs under experiment_runs/drift-recovery/{post-bank-headtohead*,h4-compound}/. SMOKE (floor-tier): python -m unittest test_post_bank_head_to_head -> OK (pre-run); the campaign itself is the end-to-end evidence. VERDICT (honest negative): LIVING BANK WINS = False on both datasets. The gate is C5 (living/annealed bank) must beat BOTH C5s (frozen static bank) AND C5r (one-shot router). - SciFact: C5 == C5s 0.131135 (bit-identical); C5r one-shot 0.180862 beats living. - NFCorpus: C5 == C5s 0.046389; living edges C5r 0.045817 but ties static. The living/annealed lifecycle adds nothing over a frozen static bank; a one-shot router is competitive-or-better. Living bank NON-PROMOTED (docs/next-session.md Disposition). C5s is the honest post-bank baseline if any is kept. H4 (single-seed SciFact C4a seed 42): compound_cycles True 0.005258 vs False 0.236297 (~45x collapse). Compounding is a rejected design; idempotent one-shot fixed point is correct. Quantifies the paper Section 5.2 overshoot note. ## Brutal Honesty - No new L1/L4/L8: no production code changed; runs merged drivers + records results. - L5 (disclosed): H4 is a single-seed (42) ablation; direction is unambiguous (~45x) but the exact magnitude is one seed, stated in the H4 doc. - Manifests-only tracking (consistent with the L1 hygiene decision): H5 per-run JSONs stay gitignored; the small summary manifests (which carry the doc-source rows) + the two <1 MB H4 run JSONs are the tracked evidence. - Verdict is from the real GPU run, not asserted from the driver alone; every number in the docs traces to a manifest row. BHS_SELF_DRAFT: 100 CARRY_FORWARD: none (living-bank line closed as non-promoted, not debt) DEFERRED_SCOPE: L3 docs-truth sync (separate commit, same PR) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rungs Main advanced 33 commits since the vision/roadmap landed, but ROADMAP_EXECUTION.md Phase II still read as all-future work (no Status column). Syncs the roadmap + CHANGELOG to git-verified reality. EVIDENCE: per-step status verified from git log / git show on the merged PRs and targeted greps for the open steps (recorded in docs/waypoint-research-2026-06-09/panel/l3-docs-truth-draft.md). SMOKE (docs-only): no code changed; ruff/tests unaffected. Changes: - ROADMAP_EXECUTION.md Phase II table gains a Status column + a git-verified status snapshot: rungs 9-12 DONE, 13 PARTIAL (post-bank prune/re-anneal only, NOT isomer/convergence -> Evidence-DAG prune), 14 apparatus DONE with the H5 living-bank verdict a hard negative, 15-17 OPEN. - CHANGELOG: apparatus-merged bullet (rungs 10-14 PRs) + still-OPEN bullet (15/16/17); the stale "Phase II execution program" Added line corrected from "all future" to the delivered/open split. ## Brutal Honesty - No delivery claimed without a merged PR. Grep-verified the three OPEN steps are genuinely unimplemented: no PyG/DGL (15); adapter_router + QuantizationPromotionGate exist but are NOT integrated as one retrieval-fitness steering plane (16); no pool-shard parity via block_graph (17). - Step 13 is honestly marked PARTIAL, not DONE — the delivered post-bank lifecycle is not the isomer/convergence->DAG-prune the original exit criteria named. Chair must re-scope or implement before step 13 reads green. - Step 10 DONE = the rung-10 control-plane DoD, NOT a claim that historical SHIM-CD-01/02/06 rows are CLOSED (called out inline). - Step 11 schedule ownership is split across two modules (#260 engine + #280 post-bank), not the single-module monopoly the original exit text implied; marked DONE-for-lattice-rungs with that caveat inline. - Older [Unreleased] Findings (Phase I / turn 3171 / "SHIM on hold") are now stale vs lattice delivery; left in place and flagged for a separate hygiene refresh rather than silently rewritten in this commit. BHS_SELF_DRAFT: 100 CARRY_FORWARD: none DEFERRED_SCOPE: refresh of the older stale [Unreleased] Findings bullets (separate hygiene pass) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…es L1 claim) Tier B (combined review of this slice) caught that L1 (2b0fa04) claimed "only the summary manifests stay tracked / NFCorpus consistent with SciFact policy" but two #275-era logs remained tracked under experiment_runs/drift-recovery/swap-nfcorpus/: campaign.log (~191 KB) and campaign-completion.log (~115 KB). SciFact's swap tree tracks 0 logs, so those two were the last inconsistency. Untracking them here makes the L1 hygiene claim actually true at PR granularity. EVIDENCE: git ls-files experiment_runs/drift-recovery/swap/*.log -> 0 (SciFact); the two NFCorpus logs are covered by .gitignore (experiment_runs/) and will not re-appear. No other file touched. SMOKE: scripts/check_block_flag.py -> CLEAR (unchanged). ## Brutal Honesty - L13-light correction disclosed by Tier B, not self-caught in L1 - recorded here transparently rather than by silently weakening the L1 prose. - No production code; no manifests/docs/NDCG changed; only two log blobs untracked. BHS_SELF_DRAFT: 100 CARRY_FORWARD: none Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
mattmre
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This draft preservation PR retains the completed dual-Spark campaign alongside
the prior consolidated method-development history:
This is not approved for merge.
docs/next-session.mdis truthfullyBLOCKEDby three unrelated expired Carried Debt rows:CD-MLR-01,CD-R13-01, andCD-R16-01. No operator override is requested or implied.Results
Qwen-Scope
The real GBA1 smoke passed in 19.51 seconds at approximately 4.49 GiB peak RSS
and 3.62 GiB peak reserved CUDA memory. The frozen layer/tensor/SAE/top-k and
two-pass repeatability contracts passed. This proves integration only, not
model-quality, training, retrieval, steering, chelation, cost, or novelty
benefit.
RB-15
Run IDs 7 and 11 each completed 170 cells, 2,040,000 RK4 steps, and 8,332
resource checks. Both retained
FACTORIAL_KILLED_ON_FROZEN_SYNTHETIC_GATES. Mean distributed Duffing gainpassed, while interaction, worst-case excess, hysteresis, and settling failed.
RB-13 Wave 0A
SELECT and REPORT each retain 12 verified synthetic entries and verify
cross-phase:
PRW-EK0:REQUIRE_COMPATIBILITY_AND_MIGRATION_DESIGN;PRW-BIL1:REDUCE_TO_PAIRED_BOOLEAN_UNION;PRW-SPU0:ORDINARY_FACTORIZATION_EQUIVALENCE;PRW-VAR1: cross-phase null killed; route the smooth method only to thedeclared state subproblem.
The v7 SELECT attempt remains quarantined with
status=INVALID_RUNandfailure_category=RESOURCE_OR_DEADLINE; it is not a campaign result. All RB-13outcomes remain synthetic, non-independent-label evidence with scientific and
novelty claims
UNCONFIRMED.Validation
ARCHIVED_COPY_VERIFIED.credential, private user path, or local IP in the staged delta.
/healthreturns 200,/v1/modelsadvertisesdeepseek-v4-flash-0731, a real completion returnedRESTORED, and running/waiting returned to zero.BLOCKED/FAIL as designed due the pre-existing debts.Brutal Honesty
Not established: production RAG behavior, training, recall/cost/speed benefit,
causal intervention utility, held-out task improvement, or novelty. RB-15 is a
bounded source-inspired analogue. RB-13 Wave 0A does not validate the broader
evidence-kernel architecture. Archive custody roots are not signatures or host
attestation. This PR does not repair the expired repository-level debt.
Visibility: hidden method-development code, protocols, and artifacts only; no
production UI/API/release activation.
EVIDENCE: retained under
artifacts/method-dev/qwen-scope-real-spark-smoke/,artifacts/method-dev/rb13-wave0a/, andartifacts/method-dev/rb15-distribution-nonlinearity-factorial/.SMOKE: real pinned Qwen/SAE run PASS; exact DeepSeek service restoration PASS; no production CHELATEDAI utility smoke claimed.
BHS_SELF_DRAFT: 100
BHS_SELF_DRAFT_AGENT: /root
BHS_TIER_B: 100
BHS_TIER_B_AGENT: /root/spark_campaign_release_review
BHS_TIER_B_SEVERITY: none
BHS_OFFICIAL: 100
CARRY_FORWARD: CD-MLR-01, CD-R13-01, CD-R16-01; resolve before merge, then preregister a held-out real-model causal intervention with matched controls.
DEFERRED_SCOPE: production utility, causal benefit, and novelty evaluation.
LOOP_ITERATIONS: 3
OPERATOR_OVERRIDE: none