[Whisper] Fix whisper tokenizer #34537
eustlb
merged
27 commits into
huggingface:main
from
eustlb:fix-whispertokenizer
handle single timestamp ending
751f88ab
include last timestamp token
000dccdb
handle single timestamp ending
70c8aaca
Merge branch 'huggingface:main' into fix-whispertokenizer
3f9f7eab
eustlb
changed the title
eustlb
changed the title 
avoid floating points arithm limitations
e7532066
ensure float64 operations
c53fb2ce
new test
185fb555
make fixup
429904c0
make copies
1c722443
Merge branch 'main' into fix-whispertokenizer
ad4f3553
eustlb
changed the title
Merge branch 'main' into fix-whispertokenizer
09af9de0
handle edge case double tokens ending with different tokens
7d6f9b47
handle single timestamp ending
937cd2a1
make fixup
9b1d51ed
handle conditioning on prev segments
a3cbe9f4
fix
7c0da36b
ylacombe
approved these changes
on 2024-11-21
Update src/transformers/models/whisper/generation_whisper.py
4a21249a
Merge branch 'main' into fix-whispertokenizer
5b195a88
[run-slow] whisper
e8f2f690
Merge branch 'main' into fix-whispertokenizer
d71d40a2
Merge branch 'main' into fix-whispertokenizer
961d5f69
Merge branch 'main' into fix-whispertokenizer
e04aa926
ArthurZucker
approved these changes
on 2024-12-04
Merge branch 'main' into fix-whispertokenizer
69abfe07
Merge branch 'main' into fix-whispertokenizer
16b1ecb5
don't call item() to avoid unnecessary sync
5fba3e0f
Merge branch 'main' into fix-whispertokenizer
9624ce9f
fix
88587bb5
alubbe
commented
on 2024-12-10
Reviewers
Assignees
No one assigned
Labels
run-slow
Milestone
No milestone
Login to write a write a comment.
Login via GitHub