transformers
[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix
#29453

Merged

[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix #29453

ArthurZucker merged 4 commits into main from fix-llama-decode

nit

7c817268

ArthurZucker changed the title ~~nit~~ [ `TokenizationLlama`] fix the way we convert tokens to strings to not always strip prefix space 2 years ago

Merge branch 'main' of github.com:huggingface/transformers into fix-l…

d23128dd

update test and fix test

fd3a2443

fixup

de2ba082

ArthurZucker marked this pull request as ready for review 2 years ago

ArthurZucker changed the title ~~[ `TokenizationLlama`] fix the way we convert tokens to strings to not always strip prefix space~~ [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 2 years ago

ArthurZucker requested a review from

LysandreJik 2 years ago

ArthurZucker requested a review from

amyeroberts 2 years ago

amyeroberts approved these changes on 2024-03-22

ArthurZucker changed the title ~~[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces~~ [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix 2 years ago

ArthurZucker merged a2a7f716 into main 2 years ago

ArthurZucker deleted the fix-llama-decode branch 2 years ago

Reviewers

amyeroberts

LysandreJik

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix #29453 Merged

[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix #29453

transformers
[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix
#29453

Merged