transformers
Fix StopStringCriteria to handle tokens above len(tokenizer)
#35797
Merged
Merged
Fix StopStringCriteria to handle tokens above len(tokenizer) #35797
Rocketknight1
merged
8 commits into
main
from
fix-stop-string-criteria-vocab-size
zucchini-nlp
approved these changes
on 2025-02-06
Fix StopStringCriteria to handle tokens above len(tokenizer)
ef03b4c8
Use self.stop_strings instead of stop_strings
647d0db8
Handle clipping correctly
b8e17cbd
make fixup
f690398b
Update test to the new embedding vecs
097c57de
Use much bigger values in the mismatch test
cb8a0f93
Typo fix
15ff9efc
Slight simplification
a322e8d5
Login to write a write a comment.
Login via GitHub