Fix repeated gradient scaling across pipeline stages (#8154)
## Summary
Fix repeated gradient accumulation scaling when pipeline parallelism is
combined with gradient accumulation.
Every pipeline stage registers an output backward hook that scales
gradients by `gradient_accumulation_steps`. The last stage therefore
scales the gradient correctly before sending it upstream, but each
preceding stage scales the already-scaled gradient again.
Fixes #8152
## Changes
- Apply gradient accumulation scaling only at the last pipeline stage.
- Preserve the existing behavior for single-stage pipelines and
`gradient_accumulation_steps=1`.
- Add a deterministic two-stage regression test that verifies both
stages receive correctly averaged gradients.
- Disable gradient clipping in the regression test so it cannot mask the
scaling difference.
## Testing
- `DS_ACCELERATOR=cpu LOCAL_SIZE=2 OMP_NUM_THREADS=1 pytest -q
tests/unit/runtime/pipe/test_pipe.py::TestPipeGradientAccumulationScaling::test_gradients_are_scaled_once`
- `pre-commit run --files deepspeed/runtime/pipe/engine.py
tests/unit/runtime/pipe/test_pipe.py`
- Two-process CPU/Gloo numerical comparison for
`gradient_accumulation_steps=1,2,8`
Signed-off-by: Yuchen Fan <functionhx@gmail.com>
