transformers
7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)

Commit

2 years ago

fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796) * add ' ' replacement for add_prefix_space * add fast tokenizer test

References

#23796 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility

Author

Connor Henderson

Parents

84bac652

transformers 7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)

transformers
7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)