onnxruntime
3d51a42f - Preserve histogram probability mass in entropy calibration (#34180)

Commit
1 day ago
Preserve histogram probability mass in entropy calibration (#34180) ### Description Correct reconstruction of a quantized histogram when evaluating entropy-calibration thresholds: - Store reconstructed bucket averages in floating point instead of truncating them into the integer histogram dtype. - Expand the final quantized bucket over its complete interval, including the remainder bins already accumulated into that bucket. - Distribute each bucket's mass only over the originally occupied histogram bins. These changes make the reconstructed distribution preserve the quantized bucket mass and original support. Candidate threshold enumeration, outlier handling, smoothing, KL selection, and threshold dtypes retain their existing behavior. ### Motivation and Context Integer truncation and incomplete expansion can change the distribution compared by KL divergence, selecting an incorrect clipping interval. The regression cases use independent Python bucket reconstruction to check mass conservation and support, then exercise real CPU calibration, persisted calibration data, static quantization, and quantized inference. For the fixed integration fixture, correcting histogram reconstruction changes the stored UINT8 activation scale from `16/255` to the independently expected `13/255`. This is a correctness result for that fixture, not a claim of a general model-accuracy gain. ### Validation - Built the Release CPU runtime from this exact PR commit using the official `tools/ci_build/build.py` on Ubuntu 24.04 / Python 3.12. The [source-build job](https://github.com/Afloat16/onnxruntime/actions/runs/43299831579/job/131467712087) completed compilation and produced the wheel. - The [successful native validation](https://github.com/Afloat16/onnxruntime/actions/runs/43304990254) consumes that exact source-built wheel and checks its source commit and installed calibration source bytes: **27 passed, 17 subtests passed**, including the existing calibration module. - Restoring the original calibration implementation in the same runtime reproduces **1 failing test method and 10 failing subtests**; the other controls pass. The follow-up validation corrects source-package shadowing during the first job's pytest collection. - Ruff lint, Ruff formatting, and `git diff --check` pass. Validation covers the CPU calibration/quantization paths above; the full C++ and GPU suites were not run.
Author
Parents
Loading