pytorch
b45880c5 - Optionally ignore utf-8 decoding error when converting std::string to python str. (#97282)

Commit
1 year ago
Optionally ignore utf-8 decoding error when converting std::string to python str. (#97282) Summary: When language models use c++ tokenizer, outputs are a c++ strings that are not necessarily valid utf-8 encodings. Default pybind11 casting uses strict utf-8 decoding. We relax the decoding using 'ignore' argument. Test Plan: https://www.internalfb.com/intern/testinfra/testrun/6473924609918070 Reviewed By: Nayef211 Differential Revision: D43970697 Pull Request resolved: https://github.com/pytorch/pytorch/pull/97282 Approved by: https://github.com/davidberard98
Author
Committer
Parents
Loading