PR #36555 Fix edge case for tokenize (#36277)

Fix edge case for tokenize (#36277) #36555

ArthurZucker merged 5 commits into huggingface:main from takina0802:tokenize-dtype

github-actions marked this pull request as draft 1 year ago

takina0802 marked this pull request as ready for review 1 year ago

Rocketknight1 force pushed from 3a1a52e1 to f61631a2 1 year ago

ArthurZucker approved these changes on 2025-05-29

takina0802 force pushed from 9f993945 to 976d550a 1 year ago

takina0802 force pushed from 976d550a to 668908df 1 year ago

takina0802 force pushed from 668908df to e186785a 1 year ago

takina0802 force pushed from e186785a to be38ef51 1 year ago

takina0802 force pushed from be38ef51 to 9a3493d0 1 year ago

takina0802 force pushed from 9a3493d0 to 9e5d2b67 1 year ago

takina0802 force pushed from 9e5d2b67 to 3d111b96 1 year ago

takina0802 force pushed from 3d111b96 to 4b48bd1d 1 year ago

takina0802 force pushed from 4b48bd1d to d1f6110d 1 year ago

takina0802 force pushed from 31cc5835 to bcfef38a 1 year ago

Fix edge case for tokenize (#36277)

89586d66

Fix tokenizing dtype for float input cases

adce3e88

add test for empty input string

080674ba

deal empty list of list like [[]]

4514fd77

takina0802 force pushed from bcfef38a to 134f6e23 1 year ago

add tests for tokenizer for models with input that is not plain text

6033faf7

takina0802 force pushed from 134f6e23 to 6033faf7 1 year ago

ArthurZucker approved these changes on 2025-08-30

ArthurZucker merged de01a22a into main 1 year ago

takina0802 deleted the tokenize-dtype branch 180 days ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone