Fix prevent duplicate GPU usage in distributed processing (#3526)
* check if num_extrs>0 and test
* test pass
* test passes
* make quality fix
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>