[`LlamaTokenizerFast`] Refactor default llama #28881
ArthurZucker
merged
28 commits into
main
from
refactor-default-llama
push legacy to fast as well
689c9aa9
super strange
7a3ddf42
ArthurZucker
commented
on 2024-02-20
Update src/transformers/convert_slow_tokenizer.py
d754f3b3
make sure we are BC
e1ef2d3e
fix Llama test
0897c61e
nit
b188318b
revert
f7e8b06b
more test
d9393584
Merge branch 'main' of github.com:huggingface/transformers into refac…
5c48f71a
style
ce042c6a
update
bbd26b00
small update w.r.t tokenizers
84d406c2
Merge branch 'main' of github.com:huggingface/transformers into refac…
038869d4
nit
6fbb0aca
don't split
6bd696e1
lol
436335dd
add a test for `add_prefix_space=False`
afa51b18
LysandreJik
approved these changes
on 2024-04-23
fix gemma tokenizer as well
312cc1d0
update
feeec97b
fix gemma
e7953f5a
nicer failures
ca25a81f
fixup
7bfe5777
pcuenca
commented
on 2024-04-23
update
05cd744f
fix the example for legacy = False
9c927419
use `huggyllama/llama-7b` for the PR doctest
187efadf
nit
9eac9c7e
use from_slow
4c40705e
fix llama
e775b7d3
Login to write a write a comment.
Login via GitHub