fix: preserve non-RL signals in zero-advantage rollouts - #3223
Open
JayZenith wants to merge 1 commit into
Open
Conversation
JayZenith
force-pushed
the
fix/preserve-non-rl-zero-advantage
branch
from
August 8, 2026 21:43
89df621 to
66d73d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ZeroAdvantageFilterdrops rollouts with all-zero GRPO advantages even when they still contain valid non-RL supervision.For example, ECHO can still have CE signal on observation tokens when an equal-reward GRPO group produces zero advantages. The same applies to hybrid RL + ref-KL cases.
Fix
Preserve zero-advantage rollouts whenever they contain nonzero non-RL loss signal through
ce_weightsorref_kl_weights.Zero-advantage rollouts with no other training signal are still filtered.
Tests
Added regression coverage for:
53 tests pass, along with Ruff and
git diff --check.