pytorch

3df1e86d - Use interactive_embedded_interpreter to dynamicly loading various third-party libraries (#66512)

Use interactive_embedded_interpreter to dynamicly loading various third-party libraries (#66512) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/66512 TLDR, we are able to use the interactive_embedded_interpreter (basically just torch::deploy interpreter with an interactive shell) to dynamicly load various third party libraries. We use the popular libraries numpy, scipy, regex, pandas for illustration purpose. A couple of changes need to be done for the interactive_embedded_interpreter: 1, we need link with :embedded_interpreter_all rather than :embedded_interpreter so we can enable DEEPBIND and use our custom loader 2, we provide a pylibRoot path to construct the InterpreterManager. The path will be added to the embedded interpreter's sys.path. Typically we can pass in the python library root path in a conda environment so torch::deploy interpreter can find all installed packages. 3, we allow interactive_embedded_interpreter execute a script to ease recording the exploration of various python libraries. ghstack-source-id: 140453213 Test Plan: Install numpy, scipy, regex, pandas in the conda environment or on the machine directly. Suppose /home/shunting/.local/lib/python3.8/site-packages/ is the root path for the installed libraries. - buck run mode/opt :interactive_embedded_interpreter -- --pylib_root=/home/shunting/.local/lib/python3.8/site-packages/ --pyscript=~/p7/iei_examples/try_regex.py content of try_regex.py: ``` import regex print(regex) pat = r'(.+)\1' print(regex.match(pat, "abcabc")) print(regex.match(pat, "abcba")) print("bye") ``` - buck run mode/opt :interactive_embedded_interpreter -- --pylib_root=/home/shunting/.local/lib/python3.8/site-packages/ --pyscript=~/p7/iei_examples/try_numpy.py content of try_numpy.py: ``` import numpy as np print(f"numpy at {np}") a = np.random.rand(2, 3) b = np.random.rand(3, 2) print(np.matmul(a, b)) ``` - buck run mode/opt :interactive_embedded_interpreter -- --pylib_root=/home/shunting/.local/lib/python3.8/site-packages/ --pyscript=~/p7/iei_examples/try_scipy.py content of try_scipy.py: ``` import numpy as np from scipy import linalg mat_a = np.array([[1, 0, 0, 0], [1, 1, 0, 0], [1, 2, 1, 0], [1, 3, 3, 1]]) mat_b = linalg.inv(mat_a) print(mat_b) ``` - buck run mode/opt :interactive_embedded_interpreter -- --pylib_root=/home/shunting/.local/lib/python3.8/site-packages/ --pyscript=~/p7/iei_examples/try_pandas.py content of try_pandas.py: ``` import pandas as pd print(f"pandas at {pd}") df = pd.DataFrame({ "col1": [1, 2, 3, 4], "col2": [2, 4, 8, 16], }) print(df) ``` Reviewed By: suo Differential Revision: D31587278 fbshipit-source-id: c0b031c1fa71a77cdfeba1d04514f83127f79012