pytorch

eae89138 - [torch export][serialize] create a more compact stacktrace format for serialization (#121675)

[torch export][serialize] create a more compact stacktrace format for serialization (#121675) Summary: - we want fx nodes' stack trace format to be backward compatible and same as before in the program we export - however in the serialized format, we would want to show a more compact stack_trace format, otherwise the nodes attributes are dominated by stack traces - the diff implements the minimal in serialization process to dedupe node stack traces by resorting to a fileinfo_list and a filename_to_abbrev map, so we can use index to represent filenames, use lineno to represent lines. Test Plan: # llm base on D54497918 ``` buck2 run @//mode/dev-nosan fbcode//executorch/examples/models/llama2:export_llama -- -c ~/stories110M.pt -p ~/params.json ``` set up breakpoint after serialization/deserialization - serialize ``` (Pdb) v_meta = [n.meta for n in exported_program.graph_module.graph.nodes] (Pdb) paste_client.create_phabricator_paste_object(paste_creation_client_id=1093956601162697, content=str(v_meta)).number 1193647450 (Pdb) json_program = json.dumps(_dataclass_to_dict(serialized_graph.co_fileinfo_ordered_list),cls=EnumEncoder) (Pdb) json_bytes = json_program.encode('utf-8') (Pdb) paste_client.create_phabricator_paste_object(paste_creation_client_id=1093956601162697, content=str(json_bytes)).number 1193604333 (Pdb) sys.getsizeof(json_bytes) 3846 (Pdb) compressed_bytes = zstd.ZstdCompressor().compress(json_bytes) (Pdb) sys.getsizeof(compressed_bytes) 1139 ``` in P1193647450 (before serialization), search for `stack_trace` in P1193604333 (after serialization), search for `stack_trace` and `co_fileinfo_ordered_list` [note: didn't do compression in this diff since the size is pretty small and it adds complexity if we do compression] - deserialize ``` (Pdb) v_meta = [n.meta for n in deserialized_exported_program.graph_module.graph.nodes] (Pdb) paste_client.create_phabricator_paste_object(paste_creation_client_id=1093956601162697, content=str(v_meta)).number 1193629435 ``` in P1193629435, search for `stack_trace` # ads Differential Revision: D54654443 Pull Request resolved: https://github.com/pytorch/pytorch/pull/121675 Approved by: https://github.com/angelayi