fix bad performance in ApplyVisitors (#2501)
* fix bad performance in ApplyVisitors
remove all allocations during visiting
rust magic with lifetimes
use binary search instead of hash map
sort visitors initially
* add some comments about the invariant
