Fix TfIdfVectorizer weight indexing semantics (#31649)
This pull request refactors the `TfIdfVectorizer` implementation to
improve the handling of n-gram index mapping, especially for cases where
n-gram indexes are not sequential or are permuted. The changes also
update the weighting callback signature to provide more context, and add
new tests to ensure correct behavior for permuted n-gram indexes.
### Refactoring and API Changes
* Updated the signature of the `fn_weight` callback in
`TfIdfVectorizer::ComputeImpl` to take both the n-gram id and output
index, allowing for more flexible and accurate mapping between n-gram
ids and output vector indices.
* Modified all usages of `fn_weight` in `ComputeImpl` to pass both the
n-gram id and output index, ensuring the callback has the necessary
information to apply weights correctly.
[[1]](diffhunk://#diff-e352d009c96e11b341df0ca79aee9195c880751f20faf70a21cf500281d6aad3L297-R299)
[[2]](diffhunk://#diff-e352d009c96e11b341df0ca79aee9195c880751f20faf70a21cf500281d6aad3L315-R318)
* Updated the construction of `fn_weight` in the main `Compute` method
to match the new signature, and to use the correct values for output
assignment based on n-gram id and output index.
### Testing Improvements
* Added new unit tests (`Int64_IDFWeights_PermutedNgramIndexes` and
`Int64_TFIDFWeights_PermutedNgramIndexes`) to verify correct behavior
when n-gram indexes are permuted, ensuring the output vector is
populated at the correct index regardless of n-gram id order.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>