onnxruntime
585d6a68 - Validate Whisper timestamp token bounds (#33824)

Commit
12 days ago
Validate Whisper timestamp token bounds (#33824) This pull request strengthens input validation for Whisper model token IDs and improves test coverage for these checks. The main focus is on ensuring that special token IDs (such as EOS and timestamp tokens) are within valid vocabulary bounds, preventing out-of-range errors during generation. Additionally, new unit tests are added to verify these validations. **Validation Improvements:** * Added explicit checks in `BeamSearchParameters::ValidateWhisperTimestampTokenId` and `TimestampLogitsProcessor::Process` to ensure that `eos_token_id`, `end_of_text_token_id`, and timestamp token IDs are within the valid vocabulary range, throwing informative errors if not. [[1]](diffhunk://#diff-5849847bc23361fe9582a0dad2a33e947961eba317a90bb0b1125f3cdecc49b4R165-R167) [[2]](diffhunk://#diff-4c14c3eea859e0355f15377e8db7e7178ae6d4ae38bf04f2611e7910b3719146R185-R200) **Test Enhancements:** * Introduced new unit tests in `beam_search_test.cc` to verify that invalid token IDs are correctly rejected and valid boundary values are accepted, including tests for out-of-range EOS and timestamp token IDs. * Added a `TestSequences` helper class and `MakeTimestampProcessor` factory function to facilitate testing of the logits processor. * Included necessary headers for new tests and processor usage. [[1]](diffhunk://#diff-782b3e352d8957cd1226d1b87f2127c58efd5ba7a6472528afbdd6baa8232198R4) [[2]](diffhunk://#diff-782b3e352d8957cd1226d1b87f2127c58efd5ba7a6472528afbdd6baa8232198R19)
Author
Parents
Loading