transformers

78a4dafe - Fix flaky `test_causal_lm_loss_honors_shift_labels` (#50533)

Fix flaky `test_causal_lm_loss_honors_shift_labels` (#50533) * Fix flaky `test_causal_lm_loss_honors_shift_labels` The first check compared the loss for two random 0/1 `shift_labels` tensors and required them to differ. On tiny random models with nearly flat logits (e.g. Inkling, which divides its hidden states by its muP width multiplier), the per-token loss changes have random signs and can cancel out in the mean below float32 resolution, so the two losses are bit-identical and the test wrongly reports that `shift_labels` is ignored (~1 in 700-2000 runs for Inkling). Use the most and the least likely token at every position as the two `shift_labels` instead: every token loss then moves in the same direction, by the max - min spread of its logits, so the mean is bound to move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address review: probe logits without labels, drop skip guard - Reuse the existing no-labels probe forward (moved out of the `else` branch) to pick the most / least likely tokens, instead of an extra forward with labels. - Drop the skip on missing / misaligned logits: the loss would already fail on misaligned targets, and a head without logits should error. - Shorten the comment and point to #50533. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Select the logits the loss is computed on before picking `shift_labels` Git's logits also cover the image patches placed before the text, but its loss only uses the text positions, so `shift_labels` picked from all of the logits had the wrong length. Add a `_logits_scored_by_loss` hook (identity by default) and override it in `GitModelTest` to drop the image patch positions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>