transformers
Fix for slow the bug tokenizer adding spaces to single id decodes
#32564
Merged
Merged
Fix for slow the bug tokenizer adding spaces to single id decodes #32564
itazap
merged
25 commits into
huggingface:main
from
DuyguA:fix/tokenizer-decoding-space
_decode signature change and quick return
74da1b46
added bunch of decoding tests
6f4c1e6e
signature match and return
7ddc3caa
added tests for decoding
15c2d9ea
DuyguA
changed the title 
merged decoding test
3716cfde
more tests for special tokens
251a5ac0
cosmetics
2e82e67d
fixed param
97d5cb1c
ruffed the file
f5da92b4
itazap
commented
on 2024-08-29
itazap
commented
on 2024-08-29
refinement for single special tokens
2b097ebd
added test for single special tokens
420993c7
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
689b93e5
itazap
approved these changes
on 2024-09-09
itazap
commented
on 2024-09-10
itazap
commented
on 2024-09-10
itazap
commented
on 2024-09-10
itazap
commented
on 2024-09-10
itazap
commented
on 2024-09-10
slight change to test name
8a29adf1
itazap
commented
on 2024-09-10
minor change test name for skip tokens
0d858cdb
killed already defined var
7ed431e1
minor update with vars
0c812cc2
killed already defined var once more
29c59501
itazap
approved these changes
on 2024-09-10
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
edca13e4
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
a0227b10
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
7a2a9c1b
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
7373ab77
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
ffbfd050
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
2362fe60
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
eefe4b28
Merge branch 'huggingface:main' into fix/tokenizer-decoding-space
1051d1c9
Login to write a write a comment.
Login via GitHub