optimum-intel
2be2e75a - Add OpenVINO Tokenizers (#513)

Commit
1 year ago
Add OpenVINO Tokenizers (#513) * Convert tokenizers with openvino_tokenizers * Update optimum/exporters/openvino/__main__.py * Refactor and Add Tests * Fix t5 Test * Add Warning * Return Tests * Move export_tokenizer to convert.py Reuse existing preprocessors * Avoid Double Tokenizer Save * Fix Style * Refactor After Review * Skip Tokenizers Tests If No Package Installed Check logs from tokneizers test * Style Fix * Fix OV Tokenizers Check * Fix Tests * Add Missing return * Turn off tokenizer message if not installed * Move tokenizers to OV dependencies * Check OV Compatibility * Bump OV Version * Move OpenVINO Tokenizers To Optional Dependencies * Add --convert-tokenizer Option to CLI * Fix SD Tokenizer --------- Co-authored-by: Sergey Lyalin <sergey.lyalin@intel.com>
Author
Parents
Loading