benchmark
bd8759d5 - Seed the recorded import aliases and builtins-dict key into the guard scope on load (#196817)

Commit
64 days ago
Seed the recorded import aliases and builtins-dict key into the guard scope on load (#196817) Summary: When Dynamo traces a function it mints synthetic names into the TRACING process's `fn.__globals__` and roots guards at them. Two kinds matter here: ```python class M(torch.nn.Module): def __init__(self): super().__init__(); self.lin = torch.nn.Linear(4, 4) def forward(self, x): if not isinstance(x, torch.Tensor): # a builtin reference raise TypeError(type(x)) return self.lin(x) # a child-module call torch.compile(M(), backend="eager")(torch.randn(2, 4)) g = M.forward.__globals__ [k for k in g if k.startswith("__import_")] # ['__import_torch_dot_nn_dot_modules_dot_module', ...] [k for k in g if k.startswith("__builtins_dict__")] # ['__builtins_dict___0'] g["__builtins_dict___0"] is builtins.__dict__ # True: the LIVE builtins dict ``` The guards read those names through `G[...]`: ``` G['__import_torch_dot_nn_dot_modules_dot_module']._global_forward_hooks # DICT_LENGTH == 0 ___check_obj_id(G['__builtins_dict___0']['isinstance'], 140012...) # BUILTIN_MATCH, an ID_MATCH into the dict ``` A process that only LOADS an `aot_compile` artifact never traced, so its scope has none of those names. The default `guard_filter_fn` drops global guards wholesale, which is why this stayed hidden; keep them and the load dies: ```python compiled = torch.compile(m, fullgraph=True, backend="eager", options={"guard_filter_fn": keep_global_guards}).aot_compile(((x,), {})) compiled.save_compiled_function("m.pt") # ---- a fresh process ---- loaded = AOTCompiledFunction.deserialize(open("m.pt", "rb").read()) loaded(x) # RuntimeError: ... KeyError on G['__import_torch_dot_nn_dot_modules_dot_module'] ``` In-process tests could not see this: the capture had already leaked the aliases into the module dict, so the load found them there. ## Who needs a global guard on a loaded artifact A model that reads module-level state in `forward` is specialized on that state at trace time, and the global guards are the only record of it: ```python EPS = 1e-5 # baked into the graph as a constant USE_FUSED = True # picks a branch at trace time FREQS = torch.randn(4096, 64) # lifted as a graph input; dtype/shape/device guarded def forward(x): if USE_FUSED: ... return (x / (x.norm() + EPS)) * FREQS[:x.shape[1]] ``` Under the default filter a loading process that sets `EPS = 1e-3`, flips `USE_FUSED`, or binds a bf16 `FREQS` is served the traced graph anyway, with no error. That is the right default for an artifact shipped to a different codebase, and this commit leaves it alone. It is the wrong trade for the other common deployment, precompile as a warm cache across processes running the SAME code, where the module layout matches and there is no reason to give up the safety eager `torch.compile` has; `CompilePackage` already lives there and keeps every guard. `torch.compile` exposes the choice through `guard_filter_fn` and ships helpers for it (`keep_global_context_and_tensor_guards_unsafe`, `keep_tensor_guards`, `skip_guard_on_globals`). Before this commit the choice did not work: keep a global guard and the load crashed on the alias, or the guard resolved against the snapshot the tracing process serialized, which agrees with itself and so can never fail. An offered option has to either work or refuse. Some `G[...]` readers survive even the default filter, because `is_global` looks only at `originating_source`: a `SHAPE_ENV` guard on a dynamic-dim module-level tensor, and a `DUPLICATE_INPUT` whose `source_b` is a global. The shape-guard case raised `KeyError` on every default load before this commit, so the fix below reaches artifacts that never touched the filter. ## Seed the guard scope at load, gated per name `AOTCompiledFunction.__post_init__` now calls `_seed_guard_scope` before `load_guard_manager`, writing into whichever dict the guards will resolve against. In outline: ```python def _seed_guard_scope(guard_scope, output_graph): # aliases: the artifact recorded {alias: module_name} in runtime_env.import_sources for alias, module_name in runtime_env.import_sources.items(): if alias in output_graph.global_scope and alias not in guard_scope: guard_scope[alias] = importlib.import_module(module_name) # builtins key: only when a deserialized guard is rooted at exactly that name roots = {get_global_source_name(g.originating_source) for g in output_graph.guards} if builtins_key in roots: CleanupHook.disown(guard_scope, builtins_key) ... # the re-derive below guard_scope.setdefault("__builtins__", builtins.__dict__) guard_scope[builtins_key] = get_builtins_dict(guard_scope) ``` Each name is gated on a kept guard being rooted at THAT name, not on the artifact having some global-rooted guard, because this writes into a dict that may be a user module's live namespace and installs no `CleanupHook`. A gate that coarse pays exactly the cost it exists to avoid in the case it admits: an artifact whose only global guard is on a user global would still write the builtins key into the caller's namespace, and would `disown` a name a live compile in that module owns. The two gates differ because the serializer records the two names differently: - **Aliases** are gated on the serialized `global_scope`, which the serializer already pruned to the names the kept guards resolve at check time. That set is wider than the guards' `originating_source`s: a `DUPLICATE_INPUT` reads its `source_b` and a `SHAPE_ENV` guard reads the shape-env sources, and a scope lacking either fails with the same `KeyError on G[...]` (measured). That is exactly the set an alias must be seeded for. An alias can only be missing from a caller-supplied scope, since `GraphRuntimeEnv.forward_callable` builds `fn.__globals__` with every recorded alias already imported. - **The builtins key** cannot use that gate: the serializer writes it into the pruned scope whether or not a guard reads it. So it is matched against the roots of the deserialized guards. The two wider channels never root at the builtins key: `load_builtin_from_argval` is the only place a source under that key is minted, and `guard_on_key_order` cannot name it, since every site that fills it adds the source of a dict or set variable. That is a structural property, so `test_builtins_key_gate_covers_the_other_serializer_channels` pins it: one artifact that really populates both channels (a non-empty `shape_env_sources`, and a `DUPLICATE_INPUT` whose `source_b` is a `GlobalSource`) and neither yields the builtins key. It never loads, so it passes unchanged on the parent; it guards the serializer property the gate relies on. A binding this process already had is left alone: a wrong binding FAILS a guard rather than passing it, and a caller-supplied scope may legitimately carry these names. That deliberately differs from `install()`'s builtins branch in package.py, which raises on a mismatch. `CleanupHook.disown` still runs on the builtins key even when its value is kept: a pre-reset compile may own the name and would delete the binding when its hook is collected. `test_load_disowns_only_a_builtins_key_a_guard_reads` pins both directions against a live compile that minted the artifact's exact key name. The aliases need no `disown`: `import_source` and `CompilePackage._install_global` bind one by plain dict assignment, and `install_global_unsafe`, the only site that creates a `CleanupHook`, never installs an alias. ## The builtins key must point at LIVE builtins, not a snapshot On the DEFAULT load path the key is often already present, and that is the second bug. When the generated bytecode itself reads a builtin (one that survived into the output instead of being constant-folded away), the reconstruction goes through `DictGetItemSource(GlobalSource(key))`, so `get_runtime_env` records the key in `used_globals` holding a pickle-FILTERED COPY of the tracing builtins. `forward_callable` spreads `used_globals` into `fn.__globals__`, which on the default path IS the guard scope. The old `builtins_key not in guard_scope` gate bailed, and the guard compared identities inside a frozen snapshot: ```python def fn(x): if isinstance(x, torch.Tensor): # constant-folded; its BUILTIN_MATCH is kept via keep_builtin_guards return len # `len` survives, so the bytecode reads G['__builtins_dict___0']['len'] loaded = AOTCompiledFunction.deserialize(data) # default path real = builtins.isinstance builtins.isinstance = lambda o, t: real(o, t) # behaviorally identical wrapper loaded.guard_check(x) # parent: True (ID_MATCH against the frozen copy, which still holds the old object) # now: False (the scope holds builtins.__dict__ itself, so the rebind is seen) ``` A kept guard that cannot fail is the whole bug: the artifact stops noticing the one change it was kept to detect, on the ORDINARY load. So the key is re-derived when, and only when, the value bound there `is` the artifact's own recording (`runtime_env.used_globals.get(builtins_key)`, compared by identity). Any other binding, a caller's copy of the real builtins included, is a value this process chose and stays: ```python scope = {builtins_key: dict(builtins.__dict__)} loaded = AOTCompiledFunction.deserialize(data, guard_globals=scope) scope[builtins_key] is builtins.__dict__ # False: the caller's dict is kept "__builtins__" in scope # False: nothing extra was written ``` `test_load_rederives_a_builtins_key_bound_to_its_own_snapshot` pins both arms on an artifact whose bytecode really reads the key (it asserts the key is in `bytecode.co_names` and in `used_globals` before it goes near a load). Two consequences of the re-derive, both deliberate. First, `__builtins__` is seeded with the live `builtins.__dict__` rather than a copy, for the same identity reason: seeding `builtins.__dict__.copy()` leaves `guard_check` True after the `isinstance` rebind (measured). That branch fires only when the scope has no `__builtins__` at all, which a real module namespace always has (CPython binds it to `builtins.__dict__` itself); the scope that lacks it is `fn.__globals__`, a plain dict literal from `forward_callable`. The `builtins.__dict__.copy()` in `GuardBuilder.__init__` is not a precedent: it copies so the torch.package loop below it can write demangled module names without touching real builtins. Second, the recording was filtered for picklability alone, never narrowed to what this process has, so a builtin the tracing process had and the loading one lacks stops being readable: with a kept guard on that name the load says so, ``KeyError on G['__builtins_dict___0']['aot_probe']`` out of `guard_check`, and with that guard filtered out the read raises `KeyError` from the generated bytecode. `test_load_rederives_over_a_builtin_the_loading_process_lacks` pins both arms. Binding a dict that merges the recording over live builtins would satisfy both readers on paper and is the frozen-snapshot bug again for every shared name (measured: it takes `guard_check` after the rebind back to True). The seeding runs AFTER `forward_callable` on purpose: `PyFunction_New` caches `__builtins__` at creation, so writing `__builtins__` afterwards cannot rewire what the compiled bytecode resolves against (measured on 3.12: rebinding after creation is inert, binding before creation decides the lookup). The builtins KEY is the exception, and deliberately so: it is an ordinary global that `LOAD_GLOBAL` resolves live on every call, so the re-derive also decides what the generated bytecode subscripts, and it can only fire when that bytecode reads the key, since that read is what makes `get_runtime_env` record the copy. ## `guard_globals=`: a caller-controlled guard scope `deserialize` grows a keyword-only parameter whose semantics differ from `f_globals`: ```python # f_globals is MERGED over the scope rebuilt from the artifact: a name it omits # still resolves to the baked-in value. AOTCompiledFunction.deserialize(data, f_globals={"EPS": 1e-3}) # guard_globals REPLACES the guard scope, no fallback. A missing name fails the # guard, and an EMPTY dict is an empty scope rather than "no scope". loaded = AOTCompiledFunction.deserialize(data, guard_globals={}) loaded(x) # RuntimeError: KeyError on G['EPS'] (for an artifact whose kept guard reads EPS) # The load WRITES its seeds into the dict passed, so pass the one they should land in. scope = {"__name__": "fresh_module"} loaded = AOTCompiledFunction.deserialize(data, guard_globals=scope) builtins_key in scope, "__builtins__" in scope # (True, True) when a kept guard reads them ``` In practice the caller hands over the live namespace that defines those globals, not a hand-built dict: ```python # deployment process, in the module that binds EPS / USE_FUSED / FREQS loaded = AOTCompiledFunction.deserialize(data, guard_globals=sys.modules[__name__].__dict__) loaded(x) # refuses: the kept guard on EPS reads 1e-3 here, the graph was built for 1e-5 ``` The guards check what that module binds NOW, on every call, so a later rebind of `USE_FUSED` is caught too. The load seeds the synthetic `__import_*` and `__builtins_dict___N` names into that module dict on the caller's behalf, which is the case the per-name gating and `CleanupHook.disown` below exist for: a real namespace is being written to. A hand-built dict works as well, and an omission is loud rather than silent: ```python scope = {"EPS": EPS, "USE_FUSED": USE_FUSED} # forgot FREQS loaded = AOTCompiledFunction.deserialize(data, guard_globals=scope) loaded(x) # RuntimeError: ... KeyError on G['FREQS'] ``` There is no fallback to the baked-in value, so the failure names the missing global and the fix is adding the key; the synthetic names are never the caller's to add. Passing nothing leaves a kept global guard vacuous: the default guard scope is the reconstructed `fn.__globals__`, which holds the values recorded at trace time, so the guard compares the recording to itself. A failed guard is a refusal, since `aot_compile` has no recompile path, and that is the correct outcome: `EPS` is a constant inside the graph, so no scope can make the loaded artifact compute with `1e-3`; the artifact was built for a different configuration and says so. `f_globals` is not the tool for this: it changes what the bytecode reads and is merged into the default guard scope, so it can fail a guard but cannot change a baked constant. `torch.compiler.load_compiled_function` does not forward `guard_globals` yet, so today it is reachable only through `AOTCompiledFunction.deserialize`; the public plumbing is left for a follow-up (see the FAQ). It is held by reference and never merged into `fn.__globals__`, so a caller-supplied scope cannot rewire what the compiled bytecode reads. The load refuses a `guard_globals`/`f_globals` whose `__builtins__` is neither a dict nor a module, naming the parameter it arrived by and the offending TYPE: ```python AOTCompiledFunction.deserialize(data, guard_globals={"__builtins__": None}) # TypeError: guard_globals['__builtins__'] must be a dict or a module, got NoneType ``` The parameter name is conditional because `f_globals` is merged into `fn.__globals__` before the function is built, so the default scope can carry a bad binding too; the value is not printed because a repr on a load failure path runs user code. The check needs only `guard_scope.get("__builtins__")`, so it is hoisted above the alias writes and above `disown`, and runs whenever a kept guard roots at the key rather than only when the key is about to be derived. Otherwise a refused load has already bound every guarded alias into the caller's dict and stripped a live compile's ownership of the key, and a caller who pre-binds the key alongside a bad `__builtins__` loads successfully. The last arm of `test_load_refuses_a_non_dict_builtins_binding` pins both against a scope where a `CleanupHook` owns the key. ## Three places still resolved `G` against the serialized snapshot Re-rooting the guards at a caller's dict exposed three sinks that were not reading the runtime scope. **A falsy scope was "no scope".** `GuardBuilder.__init__` took `runtime_global_scope or global_scope`, so `guard_globals={}` was silently replaced by the globals serialized with the artifact, the baked-in resolution this path exists to forbid, and whether that happened depended on whether the seeding above had made the dict truthy. It now tests `is None`, which is what "no scope supplied" always meant: the eager path passes `None` and is unchanged, and `CompilePackage.install()` passes `sys.modules[...].__dict__`, which is never empty. `load_guard_manager` says so at its own signature, since it is the sink that receives the choice. `test_load_into_an_empty_scope_refuses_a_lifted_global` pins it: an artifact whose kept guard reads a global the graph LIFTED, loaded with `guard_globals={}`, has to refuse on `KeyError on G['EPS']`. The old fallback holds a FakeTensor for that global, so its `TENSOR_MATCH` fails on the type instead, which the regex also rejects; the fallback answers on a baked-in value only for a specialized global the serializer kept a real value for, which is the `AOT_POOL_MODE` arm of `test_load_seeds_no_name_no_guard_is_rooted_at`. **Python lambda shape guards.** A `SHAPE_ENV` guard roots at a `ShapeEnvSource`, which is not global, so the default filter KEEPS it, and with `enable_cpp_symbolic_shape_guards` off (the default) it installs as one Python lambda over its code parts, which carry a literal `G['NAME']` for a global with a dynamic dim. `add_python_lambda_leaf_guard_to_root` exec'd that lambda over the SERIALIZED `global_scope`, so it never saw `runtime_global_scope`: ```python # default filter: the global is pruned from the serialized scope, so nothing names it loaded(x) # KeyError on 'AOT_DYN_ROWS' on EVERY load, the default one included # kept global guard: the serialized scope holds a FakeTensor with the TRACED sizes loaded.guard_check(torch.randn(4, 5)) # passed against a traced (4, 3), whatever the caller's dict held ``` The lambda now closes over `runtime_global_scope`, the dict the C++ tree is rooted at; on the eager path that is `global_scope` itself, so nothing there changes. `CompilePackage.install()` gets the same fix through the same line, and there it is the silent arm: a package keeps every guard, so a shape constraint on a dynamic-dim module-level global was checked against the pickled FakeTensor and passed whatever the module bound. A one-row global was served the graph built for `2 <= rows` (measured under `fail_on_recompile`); the fix turns that stale hit into a recompile. `test_installed_shape_guard_on_a_global_reads_the_live_module_dict` pins that half through `DiskDynamoStore` save/load and `install()`, anchored on `verbose_code_parts=["2 <= G['PKG_DYN_ROWS']` because the stance message prints the whole guard tree before the parts that failed. Both arms rebind the global to a fresh tensor because the loaded `TENSOR_MATCH` rejects the marked original: the pickled FakeTensor carries no `_dynamo_dynamic_indices`, so the load guards `hasattr(G['PKG_DYN_ROWS'], '_dynamo_dynamic_indices') == False`. `test_load_resolves_a_python_shape_guard_against_the_guard_scope` pins the default-filter case and `test_load_python_shape_guard_cannot_pass_on_the_serialized_scope` the silent arm; on the parent the first fails its default load with `GuardManager check failed ... ["'AOT_DYN_ROWS'"]` and the second with `RuntimeError not raised`. **Fail reasons.** `guard_manager.global_scope` is the dict `get_guard_fail_reason_helper`, `recompilation_reason_for_no_tensor_aliasing_guard` and `guard_error_hook` evaluate failed code parts against. None is reachable from a loaded artifact today (the first two run over eager `cache_entries` and the `skip_guards_check=True` sanity check a deserialized state disables; the hook fires from `lookup_in_list` while precompile entries go through `run_root_guard_manager`), so leaving it on the serialized dict was a latent split rather than a live bug. But it was the one `G` that disagreed with both the lambda and the accessor tree, and wiring recompile reasons to precompile entries would have reintroduced what this commit closes. It is re-pointed the same way; on the eager path `guard_manager.global_scope["G"] is fn.__globals__` before and after (measured). ## Interaction with a later compile in the same module Seeding a captured `__builtins_dict___N` into a live module dict puts a name in the way of a later compile in that module, whose `unique_id` counter is still behind and will regenerate it. The mint sites below this commit already skip forward past a name that is taken (`unique_id_unbound_in`), so that load-then-compile does not trip `CleanupHook.create`'s "already exists in scope" assertion. That sequence is pinned here rather than where the skip lands, since only this commit has a load that does the seeding: the `guard_reads_builtins` arm of `test_load_disowns_only_a_builtins_key_a_guard_reads` compiles again, under a rewound counter, in a module dict where the key is already bound by the earlier live compile and the load only disowned it. ## Review order Read the guard scope `__post_init__` chooses first, then `_seed_guard_scope`, then the `is None` test in `GuardBuilder.__init__` that lets an empty scope survive the trip, then the `G` a Python lambda guard closes over in `add_python_lambda_leaf_guard_to_root` and the one `guard_manager.global_scope` keeps for fail reasons, then the tests. `test_builtins_key_gate_covers_the_other_serializer_channels` runs with `assume_static_by_default=False` so the shape-env channel carries a GLOBAL-rooted source (`AOT_DUPE_A`), asserted alongside the dupe half; without the flag it fails `'AOT_DUPE_A' not found in set()`. The no-overwrite arms of `test_load_keeps_a_prebound_builtins_scope` and `test_load_rederives_a_builtins_key_bound_to_its_own_snapshot` pre-bind through a `CleanupHook` and fire it after the load, so a `_seed_guard_scope` that never runs on a caller scope fails them with `never disowned` / `the load replaced, or never disowned, the builtins dict it was handed` (checked by mutation). X-link: https://github.com/pytorch/pytorch/pull/196817 Approved by: https://github.com/pytorchgreenlight ghstack dependencies: #196767 Reviewed By: atalman Differential Revision: D120011425 fbshipit-source-id: 8bfdcc5bb2d486679bed1617894d090a94df299e
Author
Committer
Parents
Loading