llama.cpp
7144124a - vocab : extract bpe_rank loading into function (#30962)

Commit
1 day ago
vocab : extract bpe_rank loading into function (#30962) * vocab : extract bpe_rank loading into function This commit extracts the BPE rank loading logic into a separate function named `load_bpe_ranks`. The motivation for this is that the came code is currently used by both gpt2/hybriddna/whitespace tokenizer models and gemma4 and this avoids the duplication of code to improve readability and maintainability. * change signature of load_bpe_ranks
Author
Parents
Loading