text-generation-inference

94304649 - nix: support Python tokenizer conversion in the router (#2515)

nix: support Python tokenizer conversion in the router (#2515) Ideally we wouldn't have the router wrapper that this change adds, but when I give PyO3 a Python interpreter with packages, it ends up linking libpython from the Python interpreter rather than the constructed environment and cannot pick up the Python modules as a result.