llama.cpp
Add Unigram tokenizer needed by T5 and FLAN-T5 model families
#8089
Merged
Merged
Add Unigram tokenizer needed by T5 and FLAN-T5 model families #8089
fairydreaming
merged
5 commits into
ggml-org:master
from
fairydreaming:t5-clean-2
llama : add T5 model architecture, tensors and model header parameters
c2c799ce
llama : add handling of byte tokens in UGM tokenizer (same as in SPM)
f4c03c09
ggerganov
approved these changes
on 2024-06-25
llama : replace allocated precompiled_charsmap buffer with std::vecto…
87b7dd23
Merge branch 'ggerganov:master' into t5-clean-2
21d36842
llama : fix whitespace formatting
f23ff913
ggerganov
commented
on 2024-07-02
Reviewers
Assignees
No one assigned
Labels
Review Complexity : Medium
Milestone
No milestone
Login to write a write a comment.
Login via GitHub