transformers
7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)

Commit

2 years ago

fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796) * add ' ' replacement for add_prefix_space * add fast tokenizer test

References

#23796 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility

#27720 - Add common processor tests

#29969 - [SigLIP] Add fast tokenizer

#32831 - [Docs] Update resources

#33111 - [Backbone] Remove out_features everywhere

#33174 - [Zero-shot image classification pipeline] Remove tokenizer_kwargs

#39821 - Support MetaCLIP 2

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#62 - Add initial DEIMv2 model implementation

#65 - Fix RTDetrV2 sine position embedding ordering

#43710 - [Docs] Add docs for GLM-OCR and fix EomT-DINOv3

#43729 - [Doc tests] Fix bug

Author

Connor Henderson

Parents

84bac652

transformers 7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)

transformers
7adce8b5 - fix: Replace `add_prefix_space` in `get_prompt_ids` with manual space for FastTokenizer compatibility (#23796)