[Futures] enhance error handling in then() (#54475)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54475
Implements the proposal in https://github.com/pytorch/pytorch/issues/53717#issuecomment-800545655. See that issue for more details, but at a high level:
1. markCompleted() immediately sets completed_ = true
2. Subclasses of future (such as cuda future) implement a nontrivial `postMarkCompletedHook` which may throw
3. If above error is caught and we call `setError`, setError itself will error out because completed_ = true.
To fix this, only call setError if the user-defined cb resulted in an error, otherwise, call `markCompleted` and let postMarkCompletedHook() throw and crash the program (per lw's thoughts this should be a fatal).
ghstack-source-id: 125300388
Test Plan: CI
Reviewed By: lw
Differential Revision: D27252965
fbshipit-source-id: fda41e8844104774aaf897286512d83ff06632b1
