llama.cpp
dac869b0 - conversion : fix Nemotron 3.5 Lightning layers (#27729)

Commit
1 day ago
conversion : fix Nemotron 3.5 Lightning layers (#27729) This commit contains a fix for the conversion of NVIDIA Nemotron 3.5 Lightning which currently incorrectly converts when using a transformers version later than 5.5.1. When converting using [convert](https://github.com/ggml-org/convert) the transformers version is 5.13.1 and this produces the following: ```console WARNING:gguf.gguf_writer:Duplicated key name 'nemotron_h_moe.attention.head_count_kv', overwriting it with new value [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] of type ARRAY ``` This does not happen with transformers 5.5.1. The reason seems to be that the configuration is different in later versions, for example when using 5.13.1 the configuration block looks like this: ```console transformers 5.13.1 raw has layers_block_type: True autoconfig has layers_block_type: True autoconfig layers_block_type: [ 'linear_attention', 'moe', 'linear_attention', 'moe', 'linear_attention', 'full_attention', 'moe', ... ] ``` And with 5.5.1 we get: ```console transformers 5.5.1 raw has layers_block_type: True autoconfig has layers_block_type: True autoconfig layers_block_type: [ 'mamba', 'moe', 'mamba', 'moe', 'mamba', 'attention', 'moe' ... ] ``` In our conversion script we only match for attention, not full attention which is causing this issue. With the changes in this commit the output with transformers 5.13.1 will be: ```console (venv) $ gguf-dump models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf | grep head_count_kv INFO:gguf-dump:* Loading: models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf 29: [INT32] | 52 | nemotron_h_moe.attention.head_count_kv = [0, 0, 0, 0, 0, 2, ...] ``` Resolves: https://github.com/ggml-org/llama.cpp/issues/27718 Refs: https://github.com/ggml-org/convert/actions/runs/32949047680/job/98116096069#step:5:2391
Author
Parents
Loading