llama.cpp
Add Unigram tokenizer needed by T5 and FLAN-T5 model families
#8089

Merged

Commits

llama : add T5 model architecture, tensors and model header parameters

sszymczy committed 2 years ago
llama : add handling of byte tokens in UGM tokenizer (same as in SPM)

sszymczy committed 2 years ago
llama : replace allocated precompiled_charsmap buffer with std::vector to avoid memory leak

sszymczy committed 2 years ago
Merge branch 'ggerganov:master' into t5-clean-2

fairydreaming committed 2 years ago
llama : fix whitespace formatting

sszymczy committed 2 years ago