PR #1261 Optimize added token split

Optimize added token split #1261

xenova merged 4 commits into main from optimize-added-token-split

Optimize split on added tokens

ff966bee

Add unit test for tokenizer with many added tokens

252c9cd7

Add dictionary splitter unit tests

fbbb0e23

Minor improvements

feb94916

xenova merged 75b352c6 into main 287 days ago

xenova deleted the optimize-added-token-split branch 287 days ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone