[fix] fix token healing tests and usage errors #33931
ArthurZucker
merged
16 commits into
huggingface:main
from
alpertunga-bile:fix_token_healing
gante
approved these changes
on 2024-10-07
auto-gptq requirement is removed & model is changed & tokenizer pad t…
79556f47
values func is changed with extensions & sequence key value bug is fixed
52281759
map key value check is added in ExtensionsTree
2bdb793f
empty trimmed_ids bug is fixed
3c73257c
tail_id IndexError is fixed
fe8394f5
empty trimmed_ids bug fix is updated for failed test
040c6417
too much specific case for specific tokenizer is removed
724aeec9
input_ids check is updated
41c1e129
require auto-gptq import is removed
ffe80362
key error check is changed with empty list check
372112b5
empty input_ids check is added
67fc659b
empty trimmed_ids fix is checked with numel function
d789d52a
usage change comments are added
55d51118
test changes are commented
744f3a28
comment style and quality bugs are fixed
125b7cdf
test comment style and quality bug is fixed
4a476baa
ArthurZucker
approved these changes
on 2024-10-16
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub