.circleci: Only downgrade if we have conda (#49519)
Summary:
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
Fixes #{issue number}
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49519
Reviewed By: robieta
Differential Revision: D25603779
Pulled By: seemethere
fbshipit-source-id: ca8d811925762a5a413ca906d94c974a4ac5b132
