PR #32240 #32184 save total_vocab_size

#32184 save total_vocab_size #32240

itazap merged 3 commits into main from vocab_size_32184

save total_vocab_size = vocab_size + user added tokens to speed up op…

24f2abbc

itazap marked this pull request as ready for review 1 year ago

itazap requested a review from

ArthurZucker 1 year ago

itazap requested a review from

amyeroberts 1 year ago

itazap requested a review from

LysandreJik 1 year ago

amyeroberts commented on 2024-07-29

itazap requested a review from

amyeroberts 1 year ago

ArthurZucker commented on 2024-07-30

itazap force pushed from e2aae0d6 to 24f2abbc 1 year ago

itazap requested a review from

ArthurZucker 1 year ago

ArthurZucker commented on 2024-07-31

updating length when added_tokens_decoder is set

cf00ef70

add test len(tokenizer)

00eef145

itazap requested a review from

ArthurZucker 1 year ago

ArthurZucker removed review request from

amyeroberts 1 year ago

ArthurZucker removed review request from

LysandreJik 1 year ago

ArthurZucker approved these changes on 2024-08-05

itazap merged 3d7c2f9d into main 1 year ago

itazap deleted the vocab_size_32184 branch 1 year ago

Reviewers

ArthurZucker

amyeroberts

Assignees

No one assigned

Labels

None yet

Milestone

No milestone