Address tokenizer shortcomings (#28428)
This pull request makes several improvements and bug fixes to the
`Tokenizer` operator in ONNX Runtime, focusing on more robust UTF-8
handling, improved memory allocation, and clearer documentation. The
changes ensure safer string processing, optimize memory usage, and
clarify the expected behavior and configuration of the tokenizer.
### UTF-8 Handling and Robustness
- Improved UTF-8 validation and character counting by passing a
precomputed vector of UTF-8 character lengths (`utf8_lengths`) through
tokenization functions, reducing redundant computation and ensuring
accurate processing.
[[1]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL43-R50)
[[2]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL144-R169)
[[3]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL356-R373)
[[4]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dR397-R399)
[[5]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL494-R517)
[[6]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dR534-R536)
- Enhanced error handling and progress guarantees in UTF-8 parsing: the
tokenizer now always advances at least one byte when encountering
unexpected or invalid UTF-8 sequences, preventing infinite loops.
[[1]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL214-R232)
[[2]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL424-R447)
[[3]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL547-R573)
### Memory Allocation Optimization
- Enabled use of C++17 polymorphic memory resource (PMR) monotonic
buffer allocator for token storage when supported, reducing per-token
allocation overhead and improving performance, especially on platforms
with PMR support.
[[1]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL13-R19)
[[2]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL243-R261)
### Regular Expression and RE2 Integration
- Explicitly set RE2 to UTF-8 mode when compiling separator and token
patterns, ensuring patterns are validated and matched as UTF-8, and
clarifying that RE2 syntax is used.
[[1]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dR129-R130)
[[2]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dR143-R144)
### Output Safety and Error Checking
- Replaced `assert` statements with `ORT_ENFORCE` to provide runtime
error checking for output buffer bounds, improving safety in production
builds.
[[1]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL320-R339)
[[2]](diffhunk://#diff-1dd777fbbd4857eefa4eab6b31e076be8a725835b423e708d13c8b528034d09dL338-R355)
### Documentation Improvements
- Updated operator documentation and attribute descriptions to clarify
that RE2 regular expressions are required, describe character
tokenization mode more clearly, and fix minor typos.
[[1]](diffhunk://#diff-81f57d9adc2cce94f85a2949a895b7ff82efcc13d05e23ee6567661f0fecb7c0L1957-R1957)
[[2]](diffhunk://#diff-81f57d9adc2cce94f85a2949a895b7ff82efcc13d05e23ee6567661f0fecb7c0L1967-R1968)
[[3]](diffhunk://#diff-81f57d9adc2cce94f85a2949a895b7ff82efcc13d05e23ee6567661f0fecb7c0L2002-R2016)
These changes collectively improve the correctness, safety, and clarity
of the tokenizer implementation and its documentation.
