transformers

6a056a16 - Fix missing post_processor in DebertaV2Tokenizer causing no special t… (#44570)

Fix missing post_processor in DebertaV2Tokenizer causing no special t… (#44570) * Fix missing post_processor in DebertaV2Tokenizer causing no special tokens In transformers v5, DebertaV2Tokenizer was rewritten to use TokenizersBackend but the post_processor (responsible for adding [CLS]/[SEP] tokens) was not set. This caused `add_special_tokens=True` to silently produce output without special tokens for models like microsoft/mdeberta-v3-base. * remove test_add_special_tokens * Add test for post_processor special tokens in DebertaV2Tokenizer