openvino_tokenizers

85be884a - [GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#494)

[GGUF] Create tokenizers factory for GGUF support in OpenVINO GenAI (#494) * [GGUF] Create tokenizers factory method Needed for GGUF tokenizers support Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com> * Clean-up implementation Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com> * Update src/tokenizers_factory.cpp * Update src/tokenizers_factory.cpp --------- Signed-off-by: Kazantsev, Roman <roman.kazantsev@intel.com>