Skip to content

fix: correct discriminative LR and early-exit training - #95

Merged
JulesBelveze merged 2 commits into
mainfrom
fix/dynamic-discriminative-lr
Aug 4, 2026
Merged

fix: correct discriminative LR and early-exit training#95
JulesBelveze merged 2 commits into
mainfrom
fix/dynamic-discriminative-lr

Conversation

@JulesBelveze

@JulesBelveze JulesBelveze commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

This fixes optimizer grouping across transformer architectures and corrects how DeeBERT and BERxiT train their exits.

  • infer learning-rate schedules independently for each encoder and decoder stack
  • assign embeddings a lower rate, task heads the top rate, and keep normalization weights out of weight decay
  • follow DeeBERT's two-stage setup and BERxiT's alternating classifier and LTE objectives
  • use standard ReduceLROnPlateau with a real epoch-loss monitor

The old optimizer-state remapping has been removed. Existing optimizer and scheduler state from the previous group layout needs to be restarted. Strict DeeBERT or BERxiT checkpoints from the old classifier and gate layout need a non-strict load or migration.

Validation

  • 76 deterministic tests
  • black, isort, flake8, scoped mypy, and diff checks

…ging

 - Simplified training stage options and clarified comments for better understanding of the configuration.
 - Improved logging of training loss to enhance monitoring during model training.
@JulesBelveze JulesBelveze changed the title refactor: make discriminative learning rates model-aware fix: correct discriminative LR and early-exit training Aug 3, 2026
@JulesBelveze
JulesBelveze merged commit 413c267 into main Aug 4, 2026
9 checks passed
@JulesBelveze
JulesBelveze deleted the fix/dynamic-discriminative-lr branch August 4, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant