transformers
6fb6859a - Add Kolibri 1 (#49463)

Commit
37 days ago
Add Kolibri 1 (#49463) * Add Kolibri 1 * fix(kolibri1): fall back to the fp32 router upcast when dtypes differ `torch.mm(..., out_dtype=)` rejects operands of different dtypes, e.g. fp32 activations into a bf16 router under autocast at inference. * fix(kolibri1): reject configs without a sliding window Without one, the attention inherited from EXAONE 4 applies RoPE in every layer, a model Kolibri 1 was never trained as. * fix(kolibri1): drop the softmax load-balancing loss Kolibri 1 balances experts through `e_score_correction_bias`, which training updates outside the loss. The loss inherited from Qwen3-MoE ranks experts by softmax over unbiased logits, so it balanced a routing the model does not use. Other bias-routed models (DeepSeek-V3, GLM-4-MoE, EXAONE-MoE) have no aux loss either. * docs(kolibri1): use a Kolibri checkpoint in the causal LM example * docs(kolibri1): trim the expert-parallel plan comment * test(kolibri1): add integration tests on the BF16 checkpoint * fix(kolibri1): update model name and contribution date * docs(kolibri1): note that the CUDA router fast path is for performance --------- Co-authored-by: Sascha Wirges <sascha.wirges@aleph-alpha-research.com> Co-authored-by: Eric Schreiber <eric.schreiber@aleph-alpha-research.com> Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>
Author
Parents
Loading