Skip to content

fix: preserve non-RL signals in zero-advantage rollouts - #3223

Open
JayZenith wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
JayZenith:fix/preserve-non-rl-zero-advantage
Open

fix: preserve non-RL signals in zero-advantage rollouts#3223
JayZenith wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
JayZenith:fix/preserve-non-rl-zero-advantage

Conversation

@JayZenith

@JayZenith JayZenith commented Aug 8, 2026

Copy link
Copy Markdown

Problem

ZeroAdvantageFilter drops rollouts with all-zero GRPO advantages even when they still contain valid non-RL supervision.

For example, ECHO can still have CE signal on observation tokens when an equal-reward GRPO group produces zero advantages. The same applies to hybrid RL + ref-KL cases.

Fix

Preserve zero-advantage rollouts whenever they contain nonzero non-RL loss signal through ce_weights or ref_kl_weights.

Zero-advantage rollouts with no other training signal are still filtered.

Tests

Added regression coverage for:

  • zero advantage + CE signal
  • zero advantage + no CE signal
  • nonzero advantage with and without CE
  • zero advantage + ref-KL signal

53 tests pass, along with Ruff and git diff --check.

@JayZenith
JayZenith force-pushed the fix/preserve-non-rl-zero-advantage branch from 89df621 to 66d73d8 Compare August 8, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant