transformers
Add Whole Word Masking and Padding Strategy to DataCollatorForLanguageModeling
#39485

Merged

Add Whole Word Masking and Padding Strategy to DataCollatorForLanguageModeling #39485

Rocketknight1 merged 4 commits into huggingface:main from rjgleaton:update-whole-word-masking-collator

Rocketknight1 commented on 2025-07-18

rjgleaton force pushed from 20b31655 to f1dd7528 135 days ago

rjgleaton force pushed from f1dd7528 to 8a959377 135 days ago

rjgleaton requested a review from

Rocketknight1 135 days ago

Rocketknight1 approved these changes on 2025-08-22

rjgleaton requested a review from

Rocketknight1 100 days ago

rjgleaton force pushed from 1505215d to e090bf75 94 days ago

rjgleaton force pushed from e090bf75 to f34f8bd5 93 days ago

Rocketknight1 force pushed from f34f8bd5 to 49a2cc0b 90 days ago

Rocketknight1 approved these changes on 2025-09-22

Add whole word masking

67ce8830

Vectorize whole word masking functions

21acc1bc

Unit test whole word masking

0a4bd821

Remove support for TF in whole word masking

ee360bf5

Rocketknight1 force pushed from 49a2cc0b to ee360bf5 90 days ago

Rocketknight1 merged 2b8a7e82 into main 90 days ago

Reviewers

Rocketknight1

Assignees

No one assigned

Labels

None yet

Milestone

No milestone