transformers
9ad815e4 - [`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606)

Commit

2 years ago

[`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606) * make sure eos and bos are properly handled for fast tokenizer * fix code llama as well * nits * fix the conversion script as well * fix failing test

References

#26606 - [`LlamaTokenizerFast`] Adds edge cases for the template processor

#56 - Add udop

#27720 - Add common processor tests

#29969 - [SigLIP] Add fast tokenizer

#32831 - [Docs] Update resources

#33111 - [Backbone] Remove out_features everywhere

#33174 - [Zero-shot image classification pipeline] Remove tokenizer_kwargs

#59 - Fix attention mask handling in EoMT-DINOv3 converter

#62 - Add initial DEIMv2 model implementation

#65 - Fix RTDetrV2 sine position embedding ordering

#44285 - Add VidEoMT

#44320 - Add SAM3-LiteText

#44375 - Add RF-DETR

#71 - Use Mask2Former ignore_value in mask matching and losses

#44385 - Fix make check-repo

Author

ArthurZucker

Parents

27597fea

transformers 9ad815e4 - [`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606)

transformers
9ad815e4 - [`LlamaTokenizerFast`] Adds edge cases for the template processor (#26606)