text-generation-inference
Remove the stripping of the prefix space (and any other mangling that tokenizers might do).
#1065

Merged

Remove the stripping of the prefix space (and any other mangling that tokenizers might do). #1065

Narsil merged 4 commits into main from fix_strip_prefix_space

Remove the stripping of the prefix space (and any other mangling that

76d5bbb0

Narsil force pushed from d5118f3b to 76d5bbb0 2 years ago

Narsil requested a review from

OlivierDehaene 2 years ago

Ignoring special tokens + updating 1 test case.

8a16c485

Revert test change.

6378303d

OlivierDehaene dismissed these changes on 2023-09-27

Update the tests with the breaking change.

bbafcee4

Narsil dismissed their stale review via bbafcee4 2 years ago

Narsil merged b32e9ce9 into main 2 years ago

Narsil deleted the fix_strip_prefix_space branch 2 years ago

Reviewers

OlivierDehaene

Assignees

No one assigned

Labels

None yet

Milestone

No milestone