transformers

3f03c379 - fix tiktoken convert to pass AddedToken to Tokenizer (#36566)

fix tiktoken convert to pass AddedToken to Tokenizer (#36566) * pass AddedToken to Tokenizer * ruff * handle dict for special tokens * option: test tokenizer from tiktoken same as fast * ruff * ruff