DeepSpeed

3ba3dc97 - fix(fp_quantizer): fix UB and negative shift warnings in fp_quantize_impl.cu (#7973)

fix(fp_quantizer): fix UB and negative shift warnings in fp_quantize_impl.cu (#7973) ## Summary - Fix `warning #68-D: integer conversion resulted in a change of sign` by using unsigned literal `1U` for all bit-shift expressions storing to unsigned types (`csrc/fp_quantizer/fp_quantize_impl.cu`) - Fix `warning #62-D: shift count is negative` by removing unused `mantisa_mask` dead code in `apply_dequantization` and `apply_selective_dequantization` The `_sign_mask` computation `1 << (_mantisa_bits + _exponent_bits)` in `apply_quantization` shifts a signed `int` literal by 31 bits (`_mantisa_bits=23, _exponent_bits=8`), which is undefined behavior in C++. Using `1U` makes the shift well-defined. For consistency and defensive programming, the same `1` → `1U` change is applied to all similar patterns in `round()`, `apply_dequantization`, and `apply_selective_dequantization`. The `mantisa_mask` variable in both dequantization functions was copy-pasted from the quantization function but **never used** in the dequantization code paths. Its initialization `mantisa_mask <<= (_mantisa_bits - q_mantisa_bits)` always produces a negative shift count because in these functions `_mantisa_bits` (quantized format, small: 1-7) < `q_mantisa_bits` (output format, large: 7 or 10). > **Note:** The issue suggested the template argument order in `launch_dequantization` / `launch_selective_dequantization` might be wrong, but analysis of the function body confirms the original order is correct — `quantized_bits = _mantisa_bits + _exponent_bits + 1` correctly computes the quantized format total bits, and `dst_mantisa << (q_mantisa_bits - _mantisa_bits)` correctly left-shifts the quantized mantissa into the output format's mantissa field. The warnings came solely from the unused dead code. Fixes #7971 ## Before / After <details> <summary>Before (18+ warnings)</summary> ``` $ nvcc -c fp_quantize_impl.cu -DBF16_AVAILABLE --expt-relaxed-constexpr \ -gencode arch=compute_86,code=sm_86 -std=c++17 fp_quantize_impl.cu(82): warning #68-D: integer conversion resulted in a change of sign fp_quantize_impl.cu(244): warning #62-D: shift count is negative (x9, one per template instantiation) fp_quantize_impl.cu(426): warning #62-D: shift count is negative (x9, one per template instantiation) ``` The `mantisa_mask` variable causing the shift warnings is declared but never used in either dequantization function. </details> <details> <summary>After (0 warnings from this file)</summary> ``` $ nvcc -c fp_quantize_impl.cu -DBF16_AVAILABLE --expt-relaxed-constexpr \ -gencode arch=compute_86,code=sm_86 -std=c++17 2>&1 | grep -E '#62-D|#68-D' (no output — all warnings eliminated) ``` Compilation succeeds with exit code 0. Only unrelated `#821-D` warnings remain from `memory_access_utils.h`. </details> ## Changes ### `csrc/fp_quantizer/fp_quantize_impl.cu` 1. **Line 38, 40, 42** (`round()`) — `1` → `1U`: Consistent unsigned shifts in `mantisa_mask`, `offset`, and exponent overflow check. Not UB today (`1 << 23` fits in `int`), but prevents future issues and silences potential sign-conversion warnings. 2. **Line 82** (`apply_quantization`) — `1` → `1U`: Fix actual UB — `1 << 31` on signed `int` is undefined behavior. 3. **Line 237** (`apply_dequantization`) — `1` → `1U` in `_sign_mask`: Consistent with `apply_quantization`. Not UB with current template args (`1 << 7`), but defensive. 4. **Line 416** (`apply_selective_dequantization`) — `1` → `1U` in `_sign_mask`: Same as above. 5. **Lines 243-244** — Remove unused `mantisa_mask` in `apply_dequantization`: Copy-pasted from the quantization function but never referenced in the dequantization code path. 6. **Lines 425-426** — Remove unused `mantisa_mask` in `apply_selective_dequantization`: Same dead code as above. ## Test plan - [x] `nvcc` compilation with `-DBF16_AVAILABLE` — 0 `#62-D` / `#68-D` warnings (was 18+) - [x] Verified `mantisa_mask` (no underscore prefix) is unused in both dequantization functions by grepping all occurrences — only used in `apply_quantization` (line 142) and `round` (lines 38-42) - [x] Verified template parameter order in `launch_dequantization` and `launch_selective_dequantization` is correct by tracing all usages of `_mantisa_bits`, `_exponent_bits`, `q_mantisa_bits` in function bodies - [x] All `1` → `1U` changes are semantically identical for non-negative shift counts; the only behavioral fix is line 82 where `1 << 31` was UB Signed-off-by: Cursx <674760201@qq.com>