PR #28428 Address tokenizer shortcomings

Address tokenizer shortcomings #28428

yuslepukhin merged 9 commits into main from yuslepukhin/tokenizer_fixes

Address safety concerns

555ae947

Cache utf8 lengths

11b75c98

Address testing issues, update the spec to the impl

971b1546

Merge branch 'main' into yuslepukhin/tokenizer_fixes

96d5c8af

Merge branch 'main' into yuslepukhin/tokenizer_fixes

e31e0617

yuslepukhin requested a review from

copilot-pull-request-reviewer 34 days ago

copilot-pull-request-reviewer commented on 2026-05-08

Address review comments

9772bfc3

yuslepukhin requested a review from

copilot-pull-request-reviewer 34 days ago

copilot-pull-request-reviewer commented on 2026-05-09

yuslepukhin requested a review from

tianleiwu 32 days ago

Address CI and review comments

993dd674

vraspar dismissed these changes on 2026-05-11

vraspar requested a review from

copilot-pull-request-reviewer 31 days ago

copilot-pull-request-reviewer commented on 2026-05-11

Address CI and review comments

76f0f391

yuslepukhin dismissed their stale review via 76f0f391 31 days ago

vraspar dismissed these changes on 2026-05-11

Address review comments

c8527220

yuslepukhin dismissed their stale review via c8527220 31 days ago

vraspar approved these changes on 2026-05-11

yuslepukhin enabled auto-merge (squash) 31 days ago

yuslepukhin merged 67d9400d into main 31 days ago

yuslepukhin deleted the yuslepukhin/tokenizer_fixes branch 31 days ago

Reviewers

vraspar

copilot-pull-request-reviewer

tianleiwu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

onnxruntime Address tokenizer shortcomings #28428 Merged

Address tokenizer shortcomings #28428

onnxruntime
Address tokenizer shortcomings
#28428

Merged