DeepSpeed
892ed83e - Stabilize FPDT logits loss for extreme finite logits (#9068)

Commit
4 days ago
Stabilize FPDT logits loss for extreme finite logits (#9068) `FPDT_LogitsLoss` computes negative log-likelihood from `softmax(...).log()`. For finite FP32 logits such as `[-1000, 1000]` with target 0, the target probability underflows to zero and the token loss becomes `+inf`, although the representable reference loss is approximately 2000. A zero reducer weight can then turn that value into `NaN` through `inf * 0`. This PR passes `log_softmax`'s result directly to the forward NLL calculation. This preserves finite representable log-probabilities. Signed-off-by: Masahiro Tanaka <tanaka.masahiro@gmail.com>
Author
Parents
Loading