PR #28428 Address tokenizer shortcomings

Address tokenizer shortcomings #28428

yuslepukhin merged 9 commits into main from yuslepukhin/tokenizer_fixes

Address safety concerns

555ae947

Cache utf8 lengths

11b75c98

Address testing issues, update the spec to the impl

971b1546

Merge branch 'main' into yuslepukhin/tokenizer_fixes

96d5c8af

Merge branch 'main' into yuslepukhin/tokenizer_fixes

e31e0617

yuslepukhin requested a review from

copilot-pull-request-reviewer 131 days ago

copilot-pull-request-reviewer commented on 2026-05-08

Address review comments

9772bfc3

yuslepukhin requested a review from

copilot-pull-request-reviewer 131 days ago

copilot-pull-request-reviewer commented on 2026-05-09

yuslepukhin requested a review from

tianleiwu 129 days ago

Address CI and review comments

993dd674

vraspar dismissed these changes on 2026-05-11

vraspar requested a review from

copilot-pull-request-reviewer 128 days ago

copilot-pull-request-reviewer commented on 2026-05-11

Address CI and review comments

76f0f391

yuslepukhin dismissed their stale review via 76f0f391 128 days ago

vraspar dismissed these changes on 2026-05-11

Address review comments

c8527220

yuslepukhin dismissed their stale review via c8527220 128 days ago

vraspar approved these changes on 2026-05-11

yuslepukhin enabled auto-merge (squash) 128 days ago

yuslepukhin merged 67d9400d into main 128 days ago

yuslepukhin deleted the yuslepukhin/tokenizer_fixes branch 128 days ago

Reviewers

vraspar

copilot-pull-request-reviewer

tianleiwu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

onnxruntime Address tokenizer shortcomings #28428 Merged

Address tokenizer shortcomings #28428

onnxruntime
Address tokenizer shortcomings
#28428

Merged