llama.cpp
0693dbfb - nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type (#23343)

Commit
24 days ago
nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type (#23343) * nemotron : support non-standard GQA, BPE tokenizer and domyn-small pre-type * gguf-py : write Metaspace prepend_scheme as GGUF metadata * vocab : parse metaspace prepend_scheme into enum, rename key * vocab : use metaspace prepend_scheme in detokenize, add TODOs for other tokenizers * vocab : use constant for metaspace escaped space --------- Co-authored-by: Martin Cimmino <martin.cimmino@domyn.com>
Parents
Loading