inference: remove `CachedMethodTable` (#44240)
Since we couldn't confirm any performance benefit from `CachedMethodTable`
in the current infrastructure (see the benchmark results in #44240), now
I'd like to propose to eliminate that entirely and save a bit of space.
