transformers
[`Whisper`] Add conversion script for the tokenizer
#27338

Merged

Commits

draft

ArthurZucker committed 2 years ago
updates

ArthurZucker committed 2 years ago
full conversion taken from `https://gist.github.com/xenova/a452a6474428de0182b17605a98631ee`

ArthurZucker committed 2 years ago
psuh

ArthurZucker committed 2 years ago
nits

ArthurZucker committed 2 years ago
updates

ArthurZucker committed 2 years ago
more nits

ArthurZucker committed 2 years ago
Add co author

ArthurZucker committed 2 years ago
fixup

ArthurZucker committed 2 years ago
cleanup

ArthurZucker committed 2 years ago
styling

ArthurZucker committed 2 years ago
add proper path

ArthurZucker committed 2 years ago
update

ArthurZucker committed 2 years ago
nits

ArthurZucker committed 2 years ago
don't push the exit

ArthurZucker committed 2 years ago
clean

ArthurZucker committed 2 years ago
update whisper doc

ArthurZucker committed 2 years ago
don't error out if tiktoken is not here

ArthurZucker committed 2 years ago
make sure we are BC with conversion

ArthurZucker committed 2 years ago
nit

ArthurZucker committed 2 years ago
Merge branch 'main' of github.com:huggingface/transformers into whisper-v3-nots

ArthurZucker committed 2 years ago
Update docs/source/en/model_doc/whisper.md

ArthurZucker committed 2 years ago
merge and update

ArthurZucker committed 2 years ago
update markdwon

ArthurZucker committed 2 years ago
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

ArthurZucker committed 2 years ago