Fix stale _added_tokens_encoder entries in cpmant and wav2vec2 (#47440)
* Fix stale _added_tokens_encoder entries in cpmant and wav2vec2
* Address review: drop temp var, assert on the added-token dicts directly
- wav2vec2: pop the encoder entry straight from the decoder pop
- tests: drop set(), assertNotIn on _added_tokens_encoder and the
contents of _added_tokens_decoder, compare the dicts directly