Enabling `tokenizers` upgrade. (#14941)

Commit

4 years ago

Enabling `tokenizers` upgrade. (#14941) * Enabling `tokenizers` upgrade. * Moved ugly comment. * Tokenizers==0.11.1 needs an update to keep borrow checker happy in highly contiguous calls. * Support both 0.11.1 and 0.11.0

References

#14941 - Enabling `tokenizers` upgrade.

#29969 - [SigLIP] Add fast tokenizer

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#62 - Add initial DEIMv2 model implementation

#65 - Fix RTDetrV2 sine position embedding ordering

#71 - Use Mask2Former ignore_value in mask matching and losses

#45110 - Add SAM 3.1

Author

Narsil

Parents

f8a989cf

transformers 08cb5718 - Enabling `tokenizers` upgrade. (#14941)

transformers
08cb5718 - Enabling `tokenizers` upgrade. (#14941)