bench(worldmodel): channel ablation — ax inert, ay harmful, generator at fault - #60
Merged
Merged
Conversation
… at fault Ledger node 19. Pre-registered before running: H1 (velocity is the missing variable) predicts zeroing ax costs nothing and ax alone performs like random; H2 (horizon binding) predicts both ablations degrade together. arm log(d_f/d_0) vs FULL success MPC full +0.0886 — 35.0% MPC ay_only (ax=0) +0.0971 +0.0086 36.5% MPC ax_only (ay=0) +0.0374 -0.0512 33.5% random +0.0402 -0.0483 34.5% zero +0.0341 -0.0545 37.0% ORACLE full -11.11 100.0% ORACLE ay_only -8.72 91.5% ORACLE ax_only -4.52 73.5% H1 CONFIRMED, H2 REFUTED: |A-FULL| = 0.0086 vs |B-FULL| = 0.0512, six-fold. THE PRE-REGISTERED DISCRIMINATOR RESOLVES. ax alone is worth 73.5% to an exact controller and 33.5% to MPC -- below the 37.0% of doing nothing. The authority exists; the planner cannot use it. The deficit is representational. NEW: all of MPC's harm is in ay, the channel it CAN partly see. ay_only reproduces full MPC; ax_only collapses to the zero policy. MPC only ever acts through ay, and those actions are uncorrelated with need (node 17, r = +0.04), which is strictly worse than not acting. Both channels fail differently: ax is unrepresentable (vel R^2 = 0.033), ay is representable at SNR 0.35 so its choices are noise. THE GENERATOR EXPLAINS BOTH. Finding contributed by review, verified here independently. _render_state carries velocity as a 10% modulation on beta: c1 = beta * (1 + 0.10*vel). Variance decomposition over 4000 samples gives task share 0.845% for c0(pos) and 0.0106% for c1(vel) -- pos gets ~80x the share. (Review reported 1.15% / 0.05% from a different freeze-point; magnitudes differ, conclusion identical, and both are recorded.) chi/peak_amp/offset are held per trajectory, so predicting them at t+1 is exact and free. The loss is minimised by encoding the nuisance and VICReg's variance term rewards precisely those directions. Node 18's vel R^2 = 0.033 is not a training failure -- it is a working objective behaving correctly on a generator that made the task variable 0.01% of the signal. Velocity is NOT destroyed: vel = 10*(c1/betaPower - 1) exactly, max error 2e-15. A ratio of two observed channels -- nonlinear, hence invisible to a linear probe, trivially within an MLP's reach. Nothing prevents learning it; nothing rewards it. CLOSES the representation-objective line as specified. Nodes 6-18 were measuring the wrong thing correctly. Next is a generator repair under node-14 blind discipline, preferring within-trajectory variation of chi/offset over raising the velocity coefficient, because the first changes what the objective is REWARDED for and the second is a knob that can be tuned until an arm wins. Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJJ8y1zo7WSqEmGNW8DQap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-registered before running. H1 (velocity is the missing variable): zeroing
axcosts nothing,axalone ≈ random. H2 (horizon binding): both ablations degrade together.Result
H1 confirmed, H2 refuted — |A−FULL| = 0.0086 against |B−FULL| = 0.0512, a six-fold separation.
The pre-registered discriminator resolves.
axalone is worth 73.5% to an exact controller and 33.5% to MPC — below the 37.0% of doing nothing. The authority exists; the planner cannot use it. The deficit is representational, not reachability.New: all of MPC's harm is in
ayay_onlyreproduces full MPC;ax_onlycollapses to the zero policy. So MPC only ever acts throughay, and those actions are uncorrelated with need (node 17, r = +0.04) — strictly worse than not acting.Both channels fail, differently:
axis unrepresentable (velR² = 0.033),ayis representable but at SNR 0.35, so its choices are noise.The generator explains both
Contributed by review, verified here independently.
_render_statedoes carry velocity —c1 = beta * (1 + 0.10·vel)— but as a 10% modulation onbeta = 10^offset · 21^(−chi):c0(pos)c1(vel)posgets ~80× the share. (Review reported 1.15% / 0.05% from a different freeze-point — magnitudes differ, conclusion identical; both are recorded.)And
chi/peak_amp/offsetare held per trajectory, so predicting them at t+1 is exact and free. The loss is minimised by encoding the nuisance, and VICReg's variance term rewards precisely those directions.So node 18's
velR² = 0.033 is not a training failure. It is a working objective behaving correctly on a generator that made the task variable 0.01% of the signal.Velocity is not destroyed:
vel = 10·(c1/betaPower − 1)exactly, max error 2e-15. A ratio of two observed channels — nonlinear, hence invisible to a linear probe, trivially within an MLP's reach. Nothing prevents learning it; nothing rewards it.Closes the representation-objective line
Nodes 6–18 were measuring the wrong thing correctly. No representation objective can encode a variable made 0.01% of the observed variance while paying a variance penalty for ignoring the other 99.99%.
Next: generator repair, under node-14 blind discipline
Two candidates, not equivalent:
chi/offsetvary within-trajectory — removes the free-prediction subsidy without touching velocity's coefficient. Changes what the objective is rewarded for rather than what it is shown. Preferred.Either way: specify using only oracle/random/zero, never a candidate arm; require the repaired generator to reproduce the ladder ordering (oracle > ay-oracle > ax-oracle > zero ≈ random ≈ MPC) before any arm is rerun; and add an acceptance assertion that a probe on the raw observation recovers
velabove a floor, so a future generator edit cannot silently reintroduce this.Ledger-only. Verified: smoke 2 runs, ledger and ADR gates green.