transformers
95940bf8 - Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init (#47944)

Commit
83 days ago
Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init (#47944) * Fix GatedDeltaNet A_log dtype to prevent -inf under bfloat16 init Sample A_log in float32 (instead of the module's default dtype) when initializing Qwen3_5, Qwen3_5Moe, and Qwen3Next GatedDeltaNet modules. Under bfloat16, torch.empty(...).uniform_(0, 16) can round a value to 0, and log(0) = -inf, permanently zeroing that head's decay gradient for the rest of training. Adds a regression test for each of the three affected models asserting A_log stays float32 and finite when the model is constructed with dtype=torch.bfloat16. Fixes #47831 * Simplify A_log init fix: bump uniform_ lower bound instead of float32 cast Per review feedback, avoid the -inf-on-log(0) bug by sampling A from uniform_(0.01, 16) instead of casting to float32. Drops the added regression test and dtype casting in favor of a one-line comment.
Author
Parents
Loading