transformers
[`TokenizerFast`] `can_save_slow_tokenizer` as a property for when `vocab_file`'s folder was removed
#25626
Merged
Overview
Commits
11
Changes
View On GitHub

[`TokenizerFast`] `can_save_slow_tokenizer` as a property for when `vocab_file`'s folder was removed #25626

ArthurZucker merged 11 commits into huggingface:main from ArthurZucker:nit-llama-fast
ArthurZucker
ArthurZucker pad token should be None by default
1bd6da0d
ArthurZucker fix tests
7c244b69
ArthurZucker nits
23254c58
ArthurZucker check if isfile vocabfile
83347855
ArthurZucker add warning if sp model folder was deleted
4405788d
ArthurZucker Merge branch 'nit-llama-fast' of https://github.com/arthurzucker/tran…
6911716d
HuggingFaceDocBuilderDev
ArthurZucker ArthurZucker marked this pull request as ready for review 2 years ago
ArthurZucker save SPM when missing folder for sloz
fb510692
ArthurZucker update the ` can_save_slow_tokenizer` to be a property
b04e274f
ArthurZucker first batch
8dc74a84
ArthurZucker second batch
458f9e2c
ArthurZucker missing one
bebfb9ae
ArthurZucker ArthurZucker changed the title [`TokenizerFast`] Warn when `vocab_file` folder was removed [`TokenizerFast`] `can_save_slow` as a property for when `vocab_file`'s folder was removed 2 years ago
ArthurZucker ArthurZucker changed the title [`TokenizerFast`] `can_save_slow` as a property for when `vocab_file`'s folder was removed [`TokenizerFast`] `can_save_slow_tokenizer` as a property for when `vocab_file`'s folder was removed 2 years ago
ArthurZucker ArthurZucker requested a review from amyeroberts amyeroberts 2 years ago
amyeroberts
amyeroberts approved these changes on 2023-08-30
ArthurZucker ArthurZucker merged 3b39b906 into main 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum