transformers
99bce63c - DeepSeek-V4.1: resolve the model structure findings

Commit
2 days ago
DeepSeek-V4.1: resolve the model structure findings The four attention classes are now one class that branches on the layer type, like DeepSeek-V4 does, so there is no subclass chain and no class dict any more. The model owns the only rotary embedding. It computes the RoPE of the compressed entries for each compress ratio and passes it down, instead of the compressing attention layers building a rotary module each. `set_engram_token_map` is a single method on the PreTrainedModel instead of one per model class. It still needs a linter ignore: it fills a saved buffer that the checkpoints leave out, not a hyperparameter. Also merges main, which dropped `keys_to_ignore_at_inference` from the parent config.
Author
Parents
Loading