Cache slow/disabled test files (#55682)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/55682
Fixes #55648
For now it downloads and writes the relevant files to the system's temp dir and marks it as valid for 3 hours.
Test Plan: Imported from OSS
Reviewed By: malfet, nikithamalgifb
Differential Revision: D27685616
Pulled By: driazati
fbshipit-source-id: 27469b85fe4b6b4addde6b22bf795bca3d4990ef
