PR #31254 Generation: fix handling of special tokens

Generation: fix handling of special tokens #31254

zucchini-nlp merged 8 commits into huggingface:main from zucchini-nlp:patch_special_tokens

fix special tokens in generatioon

48c7d2ea

Merge remote-tracking branch 'upstream/main' into patch_special_tokens

7bb10086

zucchini-nlp requested a review from

ArthurZucker 1 year ago

fix test

7afe6c2b

add warning

a3ea0cd8

fix the check

b17221a0

ArthurZucker approved these changes on 2024-06-05

warn once

e617d34c

CISC commented on 2024-06-05

fix

79608528

Merge branch 'main' into patch_special_tokens

14fd38cc

zucchini-nlp merged 5fabd1e8 into main 1 year ago

Reviewers

ArthurZucker

CISC

Assignees

No one assigned

Labels

None yet

Milestone

No milestone