Add Kolibri 1 (#49463)
* Add Kolibri 1
* fix(kolibri1): fall back to the fp32 router upcast when dtypes differ
`torch.mm(..., out_dtype=)` rejects operands of different dtypes, e.g.
fp32 activations into a bf16 router under autocast at inference.
* fix(kolibri1): reject configs without a sliding window
Without one, the attention inherited from EXAONE 4 applies RoPE in every
layer, a model Kolibri 1 was never trained as.
* fix(kolibri1): drop the softmax load-balancing loss
Kolibri 1 balances experts through `e_score_correction_bias`, which
training updates outside the loss. The loss inherited from Qwen3-MoE
ranks experts by softmax over unbiased logits, so it balanced a routing
the model does not use. Other bias-routed models (DeepSeek-V3, GLM-4-MoE,
EXAONE-MoE) have no aux loss either.
* docs(kolibri1): use a Kolibri checkpoint in the causal LM example
* docs(kolibri1): trim the expert-parallel plan comment
* test(kolibri1): add integration tests on the BF16 checkpoint
* fix(kolibri1): update model name and contribution date
* docs(kolibri1): note that the CUDA router fast path is for performance
---------
Co-authored-by: Sascha Wirges <sascha.wirges@aleph-alpha-research.com>
Co-authored-by: Eric Schreiber <eric.schreiber@aleph-alpha-research.com>
Co-authored-by: Cyril Vallez <cyril.vallez@huggingface.co>