build,uninstall: avoid removing directories in usr
This can cause a race if we happen to be trying to install something
else in the same directory we just removed. Very unlikely, but
unnecessary to remove them anyways (and generally was already broken for
Yggdrasil-provided binaries).
