DeepSpeed
9bf48c86 - fix(rollout): release the continuous graph across weight syncs (interim (a))

Commit
2 days ago
fix(rollout): release the continuous graph across weight syncs (interim (a)) Runguo's finding 2: a retained continuous-batching graph captures the derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat cache, GDN gates). refresh_weights drops those tensors, so a surviving graph would replay freed memory. Parameters are covered by the capture signature (in-place writes keep their addresses), but the derived caches are invisible to it, so nothing else invalidates them. Interim fix (a): refresh_weights releases _continuous_graph_state (synchronized, like shutdown) before dropping caches; the next rollout re-captures. Cost: one re-capture per sync. The end-state fix (b) -- refresh refills derived tensors in place so graphs survive by construction -- is deferred by design; remove this release when it lands. The new test pins the contract on any config (asserts the graph is released after sync, then post-sync generation matches a fresh injection). Red on the previous head (the graph survived the sync); config-dependent note: on Ada/dense/TP=1/small-batch the captured graph holds no derived tensor, which is why the hazard never surfaced there. Validated on L20 (card 4): red -> green; full rollout + module_inject 355 passed / 16 skipped / 0 failed. Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading