Skip offload tests for `ViTDet` (#25913)

Commit

2 years ago

Skip offload tests for `ViTDet` (#25913) * update * fix --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>