Fix per token atrributes bits #7749
jaime-m-p
merged
12 commits into
ggml-org:master
from
jaime-m-p:per-token-attribs
Add per token attrib enum
cec6a3bd
Using phi-3 for testing 'rstrip'
3ead1b97
bugfix: assertions, wrong special token list
33de2479
Implement 'rstrip' properly
ada961ce
Refactor + add 'jina-v2' for testing 'lstrip'
01c92291
Update phi-3 GGUF file (obsolete since 917dc8c)
8564c198
Update brute force test: testing 'lstrip' and 'rstrip'
83cac0ec
Fix previous commit
54e9f23b
Replace llama_token_type with llama_token_attribs
ac40ff0e
Rename token attributes
18f5fc76
Fix per token atrributes bits
f95c7dc7
Merge branch 'master' into per-token-attribs
6a0f3db7
ggerganov
approved these changes
on 2024-06-04
Login to write a write a comment.
Login via GitHub