PR #257 Fix padded vocab size on preprocessing scripts

Fix padded vocab size on preprocessing scripts #257

SaulLu merged 3 commits into LS/vocab_size from thomas/fix_padded_vocab_size

Add tokenizer options in preprocessing scripts

4b53f638

thomasw21 requested a review from

SaulLu 4 years ago

thomasw21 changed the base branch from main to LS/vocab_size 4 years ago

This should fix the TP issue?

ed23c0f3

Merge branch 'LS/vocab_size' into thomas/fix_padded_vocab_size

b891483d

SaulLu approved these changes on 2022-03-01

SaulLu merged f515b67f into LS/vocab_size 4 years ago

SaulLu deleted the thomas/fix_padded_vocab_size branch 4 years ago

Reviewers

SaulLu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone