[Serialization] Load only needed partial specializations
Similar as the last commit, it is unclear why we need to load all
specializations, including non-partial ones, when we have a TPL.
Reviewed as part of https://github.com/llvm/llvm-project/pull/133057
