Revert "Fix XNNPACK OSS Buck build (#94935)"
This reverts commit 9d2fddf820f8cf4273b12a8be5a556ba230c21cf.
Reverted https://github.com/pytorch/pytorch/pull/94935 on behalf of https://github.com/kit1980 due to The issue already mitigated by https://github.com/pytorch/pytorch/pull/94785
