transformers
b9042c4e - fix: Prevent AutoTokenizer type mismatch from directory name substrin… (#43791)

Commit
74 days ago
fix: Prevent AutoTokenizer type mismatch from directory name substrin… (#43791) When saving a tokenizer to a local directory and reloading it, the tokenizer type could change to an incorrect class (or fall back to TokenizersBackend) if the directory name contained a model type substring. We're removing this fallback behavior and explicitly check that `model_type` is provided
Author
Parents
Loading