transformers
Fix ByteLevel alphabet missing when Sequence pretokenizer is used
#33556

Merged

Fix ByteLevel alphabet missing when Sequence pretokenizer is used #33556

itazap merged 2 commits into huggingface:main from umarbutler:patch-2

Fix ByteLevel alphabet missing when Sequence pretokenizer is used

e8c6a406

itazap approved these changes on 2024-09-20

Fixed formatting with `ruff`.

f631ca37

itazap approved these changes on 2024-09-23

itazap merged fa0bb0fe into main 1 year ago

Reviewers

itazap

Assignees

No one assigned

Labels

None yet

Milestone

No milestone