fix(rollout): release the continuous graph across weight syncs (interim (a))
Runguo's finding 2: a retained continuous-batching graph captures the
derived tensors' addresses (TP>1 cat weights, Ampere/Hopper GLU concat
cache, GDN gates). refresh_weights drops those tensors, so a surviving
graph would replay freed memory. Parameters are covered by the capture
signature (in-place writes keep their addresses), but the derived
caches are invisible to it, so nothing else invalidates them.
Interim fix (a): refresh_weights releases _continuous_graph_state
(synchronized, like shutdown) before dropping caches; the next rollout
re-captures. Cost: one re-capture per sync. The end-state fix (b) --
refresh refills derived tensors in place so graphs survive by
construction -- is deferred by design; remove this release when it
lands.
The new test pins the contract on any config (asserts the graph is
released after sync, then post-sync generation matches a fresh
injection). Red on the previous head (the graph survived the sync);
config-dependent note: on Ada/dense/TP=1/small-batch the captured graph
holds no derived tensor, which is why the hazard never surfaced there.
Validated on L20 (card 4): red -> green; full rollout + module_inject
355 passed / 16 skipped / 0 failed.
Signed-off-by: Ma, Guokai <guokai.ma@gmail.com>