onnxruntime
f5de4c39 - Fix TfIdfVectorizer weight indexing semantics (#31649)

Commit
40 days ago
Fix TfIdfVectorizer weight indexing semantics (#31649) This pull request refactors the `TfIdfVectorizer` implementation to improve the handling of n-gram index mapping, especially for cases where n-gram indexes are not sequential or are permuted. The changes also update the weighting callback signature to provide more context, and add new tests to ensure correct behavior for permuted n-gram indexes. ### Refactoring and API Changes * Updated the signature of the `fn_weight` callback in `TfIdfVectorizer::ComputeImpl` to take both the n-gram id and output index, allowing for more flexible and accurate mapping between n-gram ids and output vector indices. * Modified all usages of `fn_weight` in `ComputeImpl` to pass both the n-gram id and output index, ensuring the callback has the necessary information to apply weights correctly. [[1]](diffhunk://#diff-e352d009c96e11b341df0ca79aee9195c880751f20faf70a21cf500281d6aad3L297-R299) [[2]](diffhunk://#diff-e352d009c96e11b341df0ca79aee9195c880751f20faf70a21cf500281d6aad3L315-R318) * Updated the construction of `fn_weight` in the main `Compute` method to match the new signature, and to use the correct values for output assignment based on n-gram id and output index. ### Testing Improvements * Added new unit tests (`Int64_IDFWeights_PermutedNgramIndexes` and `Int64_TFIDFWeights_PermutedNgramIndexes`) to verify correct behavior when n-gram indexes are permuted, ensuring the output vector is populated at the correct index regardless of n-gram id order. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading