fix(sft): drop rows with phantom media tokens; fail fast on fatal rank errors

[hubert-marek]

The failure this fixes

Five consecutive 64-GPU SFT runs (Nemotron-VL graft, 8×H200, cp=2 ulysses, seq 131k) wedged at the same training step under a fixed data seed: 62 ranks spinning at 100% GPU inside NCCL, two ranks at 0% in futex_wait, no error in the job log, and after dist_timeout a watchdog report blaming an FSDP all-gather on a healthy rank. It looked like fabric. It wasn't.

The two silent ranks (one CP pair) had hit, inside model.forward:

ValueError: Nemotron-VL image token/feature mismatch before scatter:
  img_context_token_id=18, image_tokens=8936, image_features=8935, ...

Their pack contained a text-only SWE trace in which an assistant tool-call writes a file whose docs contain the literal string <image> ({type<image>|bold}, a template-language example). The tokenizer maps that substring to the image-placeholder id — one phantom placeholder with no pixel features behind it, so the packed batch counts N+1 image tokens against N features.

clean_exit then caught the exception and, in its finally, called dist.destroy_process_group() — while the 62 peers were still blocked in a collective the dying pair would never join. The graceful shutdown hangs in ProcessGroup.shutdown(), the peers spin to the collective timeout, and the flight-recorder dump describes only victims (the two culprit ranks never reach a collective, so they write no trace). With --local-ranks-filter=0 the actual traceback existed only in the per-rank torchrun log of local ranks 2–3.

Fix 1 — drop rows with phantom media tokens (sft/data.py)

mm_token_type_ids is built from renderer-emitted PlaceholderRanges only, so a placeholder id at a type-0 position — or anywhere in a row that produced no multimodal data at all — is phantom text, never legitimate. SFTDataset._process now counts those and drops the row with a warning naming the example:

WARNING Dropping example 9690 (swe_code_agent-train-9776): 1 media placeholder
token(s) outside any renderer-emitted placeholder run (literal marker text in content)

The check is renderer-agnostic (uses the mm_token_type_id_map protocol attribute, inert for text-only renderers) and O(seq_len) per row against a 1–2 element set.

Fix 2 — fail fast on fatal rank errors (utils.py)

On the fatal path, clean_exit no longer attempts a graceful destroy_process_group(); it logs, flushes wandb, and os._exit(1). The launcher's failure propagation (torchrun / srun --kill-on-bad-exit) then tears the world down in seconds instead of dist_timeout minutes, and the surviving logs point at the rank that actually raised rather than at a victim collective. The clean path is unchanged: wandb.finish() then graceful destroy.

This applies to both the sync and async wrappers (the async comment about sys.exit being swallowed by the event loop applied to raise, and os._exit is immune to both).

Verification

• Guard tested against the actual offending corpus row: dropped with the warning above; healthy text-only, single-image, and multi-image (3-tile) rows all render unchanged.
• Before the guard, the row reproducibly wedged an 8-node run at the same step under seed 0 (four resumed runs + one from-scratch run); a seed bump moved the row and the run passed the step — confirming the row, not the step, as the trigger.
• ruff check / ruff format --check clean on both files.

Corpus census

A full sweep of the 46-subset corpus (13.1M rows) for literal media markers in any message field found 24 flagged rows, all in tool_calls arguments — the field the previous content-only sanitizer never walked. Rendering all 24 through the guard: 7 are real phantoms (2–11 phantom tokens each, all in the SWE-agent family; the guard drops every one) and 17 are benign (the marker never survives into the token stream on that render path). Every other subset is clean. So the guard costs 7 rows out of 13.1M while removing the entire crash class.

[hubert-marek]

Follow-up on root cause: the guard in this PR is a detector, and the class is better closed one layer up, in the renderer's text encoder.

<image> is an added/special token (id 18) in this tokenizer. add_special_tokens=False only suppresses BOS/EOS — it does not stop a special token from being recognized inside caller-supplied text:

"{type<image>|bold}"
  renderer today (add_special_tokens=False) → [1123, 4994, 18, 1124, 28118, 1125]
                                                          ↑ id 18, the placeholder
  with split_special_tokens=True            → [1123, 4994, 1060, 5497, 1062, 1124, 28118, 1125]
                                                          ↑ '<', 'image', '>' as ordinary text

So this is data being parsed as control — message content can synthesize a reserved token. Every _encode in deps/renderers has the same shape (qwen3_vl, glm45, glm5, deepseek_v3, hy3, gpt_oss, qwen35, kimi_k2, kimi_k25, nemotron3): self._tokenizer.encode(text, add_special_tokens=False) on untrusted content. Nemotron-VL is just the one that fails loudly, because a downstream count check compares placeholders against vision features. On a text-only model the same injection silently splices control tokens into the sequence with no error at all.

Encoding message text with split_special_tokens=True would close it everywhere at once. Renderer scaffold (<img>/</img>, chat-template markers) is unaffected: it goes through the separate emit_special path that resolves ids explicitly rather than by string matching.

Not proposing that here, since it touches nine renderers and changes tokenization for any existing corpus containing marker-like text — that needs a maintainer's call on default-on vs opt-in, and on whether checkpoints trained under current behavior matter. Raising it as a draft against the renderers repo separately; this PR stands on its own as defence-in-depth plus the fail-fast fix.

For scale: a corrected all-fields sweep of our 13.1M-row corpus found 24 rows carrying literal markers, all in assistant.tool_calls — the field the content-only sanitizer never walked. 7 of them render into real phantom tokens (1–2 each).