Revert "Reduce includes of CUDACachingAllocator.h (#97072)"
This reverts commit 1bcb88089468a6ebc667bd76256c4dd6f58b7ee3.
Reverted https://github.com/pytorch/pytorch/pull/97072 on behalf of https://github.com/weiwangmeta due to breaking internal builds
