transformers
fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility
#23796

Merged

fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility #23796

sgugger merged 2 commits into huggingface:main from connor-henderson:whisper-get-prompt-ids-prefix-space

connor-henderson marked this pull request as ready for review 2 years ago

sanchit-gandhi approved these changes on 2023-05-26

add ' ' replacement for add_prefix_space

0e9c5206

add fast tokenizer test

8866bcad

connor-henderson force pushed to 8866bcad 2 years ago

sanchit-gandhi approved these changes on 2023-05-31

sanchit-gandhi requested a review from

hollance 2 years ago

sanchit-gandhi requested a review from

sgugger 2 years ago

sgugger approved these changes on 2023-05-31

sgugger merged 7adce8b5 into main 2 years ago

Reviewers

sgugger

sanchit-gandhi

hollance

Assignees

No one assigned

Labels

None yet

Milestone

No milestone