transformers
61fa365b - Fix path traversal via vocab-file arguments in tokenizer_config.json (#46279)

Commit
44 days ago
Fix path traversal via vocab-file arguments in tokenizer_config.json (#46279) `tokenizer_config.json` is loaded into the tokenizer's `init_kwargs`, but `save_pretrained` never serializes the vocab-file arguments (`vocab_file`, `merges_file`, `tokenizer_file`, ...) — these are resolved from the repository via `cls.vocab_files_names`. When merging the repo-resolved paths back into `init_kwargs`, `_from_pretrained` only filled in a key that was absent, so a repo whose `tokenizer_config.json` sets e.g. `"vocab_file": "/etc/passwd"` (or `"../x"`) kept that value, which the tokenizer then opens verbatim. This lets a malicious model read an arbitrary local file via `AutoTokenizer.from_pretrained(...)` with no `trust_remote_code` (CWE-22). Let the repo-resolved path take precedence over a config-supplied value for these arguments, while still honoring an explicit path passed by the caller as a `from_pretrained` kwarg. Adds a regression test asserting an injected `vocab_file` is ignored on reload.
Author
Parents
Loading