Fix autotuning get_val_by_key to search all nested subdicts (#8177)
## Root cause
`get_val_by_key` (`deepspeed/autotuning/utils.py`) recurses into the
first dict-valued entry and returns its result unconditionally, even
when that subtree yields `None`:
```python
for v in d.values():
if isinstance(v, dict):
return get_val_by_key(v, k) # never tries the next subdict
```
So a key that lives in a later sibling subdict is reported absent. The
paired `set_val_by_key`, defined right below it, recurses into every
subdict with no early return, so the two helpers are meant to be
inverses but disagree: `set` reaches the key, `get` does not.
Where it bites: `ResourceManager.run_job` (`scheduler.py:98`) does `nval
= get_val_by_key(exp, key); if nval and str(nval) != "auto": ...` to
inject tuned values into an experiment's CLI args. A wrong `None` skips
the `if nval` guard, so the tuned value is silently not applied and the
run uses the user/default value instead. This is order-sensitive: it
only bites when the target key's subdict is not the first one searched.
## Fix
Continue the search on a `None` result instead of returning it, matching
`set_val_by_key`:
```python
for v in d.values():
if isinstance(v, dict):
r = get_val_by_key(v, k)
if r is not None:
return r
return None
```
Invariant restored: `get_val_by_key` walks every nested subdict, so it
finds a key wherever `set_val_by_key` can set it.
## Verification
Added `test_get_val_by_key_searches_all_nested_subdicts` in
`tests/unit/autotuning/test_autotuning.py`, covering: a key in a
non-first sibling subdict is found; a genuinely absent key still returns
`None`; a key in the first subdict is unaffected; and the getter agrees
with `set_val_by_key`. The test fails on `master` (first assert returns
`None`) and passes with this change; run on CPU against `d3265209`.
`yapf`, `flake8`, and `codespell` are clean on both changed files.
Not verified: an end-to-end autotuning session. The reachability of the
buggy branch through `run_job` is read from the source, not exercised on
live hardware, so this is scoped as a correctness fix to the helper and
its symmetry with `set_val_by_key`, not a claim that a particular tuning
run is broken.
Fixes #8176
Signed-off-by: Ehsan Barkhordar <realbarkhordar@gmail.com>
DeepSpeed
4b693658 - Fix autotuning get_val_by_key to search all nested subdicts (#8177)
