transformers

2da6655e - Fix gradient inflation when combining label smoothing with gradient accumulation (#47261)

Fix gradient inflation when combining label smoothing with gradient accumulation (#47261) * Fix gradient inflation when combining label smoothing with gradient accumulation When label_smoothing_factor > 0 and gradient_accumulation_steps > 1, gradients are scaled up by roughly gradient_accumulation_steps for any model where model_accepts_loss_kwargs is True (e.g. Llama, Qwen). The Trainer counts num_items_in_batch and, because it is not None, skips the 1/gradient_accumulation_steps normalization in training_step. But the loss is produced by LabelSmoother, which mean-reduces over the current micro-batch's active tokens and never receives num_items_in_batch. So neither path normalizes by the effective batch: the accumulated gradient becomes sum(S_i / N_i) instead of sum(S_i) / sum(N_i). The 2024 gradient-accumulation fix touched the model-internal loss path but not LabelSmoother. Pass num_items_in_batch into LabelSmoother and use it as the denominator when provided, falling back to the per-micro-batch active-token count otherwise (so the standalone LabelSmoother call is unchanged). * Address label smoother review feedback --------- Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>