Fix inverted assertion in remove_handler (#46227)
The assert condition in remove_handler checks 'handler not in
handlers', which raises AssertionError when removing a registered
handler and silently passes for unregistered ones. Invert the
condition to correctly validate that the handler exists before
removal.
Fixes #21506
Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
