transformers
08d609bf - Add tokenizers class mismatch detection between `cls` and checkpoint (#12619)
Commit
View On GitHub
Commit
4 years ago
Add tokenizers class mismatch detection between `cls` and checkpoint (#12619) * Detect mismatch by analyzing config * Fix comment * Fix import * Update src/transformers/tokenization_utils_base.py Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com> * Revise based on reviews * remove kwargs * Fix exception * Fix handling exception again * Disable mismatch test in PreTrainedTokenizerFast Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>
Parents
b4b562d8
FAQ Terms Privacy Refunds Impressum
Loading