transformers
a2a7f716 - [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix (#29453)

Commit

1 year ago

[ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix (#29453) * nit * update test and fix test * fixup

References

#29453 - [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix

Author

ArthurZucker

Parents

e677479c

transformers a2a7f716 - [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix (#29453)

transformers
a2a7f716 - [ `TokenizationLlama`] fix the way we convert tokens to strings to keep leading spaces 🚨 breaking fix (#29453)