whisper.cpp

OpenVINO support

#1037

Merged

Running Whisper inference using OpenVINO

This PR extends whisper.cpp to run the Whisper Encoder on OpenVINO supported devices such as CPU, and Intel GPUs (integrated & discrete).

I've tested this on number of platforms, including

• NUC Intel(R) Core(TM) i7-6770HQ ('Skylake' Skull Canyon NUC) running Ubuntu 22.04
• Core(TM) i7-1185G7 ('Tiger Lake' laptop) running Windows 11 Pro
• Core(TM) i7-12700 ('Alder Lake' Beast Canyon NUC) with installed Intel(R) ARC(TM) A770 discrete graphics card, running Windows 11 Pro

For each platform, the performance of using OpenVINO-based encoder gives a great boost in performance over the default encoder -- even for CPU -- and the ability to easily offload to another OpenVINO-supported device by simply specifying a different string at runtime (e.g. "CPU" --> "GPU") is very convenient.

High-level description of changes

This introduction of OpenVINO Encode support is modeled very closely to how whisper.cpp uses CoreML (this should be pretty obvious in the change-set). If the project is built with OpenVINO support, an OpenVINO-specific encoder is pulled into the build and instantiated at application startup time.

Also similar to CoreML, the models required to be present to take advantage of the OpenVINO encoder can be generated using a new python script in 'models' directory.

Just to point out -- something that does differ between CoreML and the new OpenVINO integration is how/when support is enabled at runtime. CoreML is enabled within the call to whisper_init_*. For OpenVINO, because we want the ability to specify a device string ("CPU", "GPU", etc.), I exposed a new API that is dedicated to initializing OpenVINO, given a ctx:

(in whisper.h):

    // Given a context, enable use of OpenVINO for encode inference. 
    // openvino_model_path: Optional path to OpenVINO encoder IR model. If set to nullptr,
    //                      the path will be generated from the ggml model path that was passed
    //                      in to whisper_init_from_file. For example, if 'path_model' was
    //                      "/path/to/ggml-base.en.bin", then OpenVINO IR model path will be 
    //                      assumed to be "/path/to/ggml-base.en-encoder-openvino.xml".
    // openvino_device: OpenVINO device to run inference on ("CPU", "GPU", etc.)
    // openvino_cache_dir: Optional cache directory that can speed up init time, especially for 
    //                     GPU, by caching compiled 'blobs' there.
    //                     Set to nullptr if not used.
    // Returns 1 on success. If OpenVINO is not enabled in build, this
    // simply returns 0.
    WHISPER_API int whisper_ctx_init_openvino_encoder(struct whisper_context* ctx,
        const char* openvino_model_path,
        const char* openvino_device,
        const char* openvino_cache_dir);

I'm happy to rework this if anyone has a better idea of how to enable OpenVINO support at init time.

main.cpp exposes a new parameter for user to set OpenVINO encode inference device (default is "CPU"):

...
else if (arg == "-oved" || arg == "--ov-e-device")    { params.openvino_encode_device = argv[++i]; }
...

And the new whisper_ctx_init_openvino_encoder API is called right after ctx creation:

   // whisper init

    struct whisper_context * ctx = whisper_init_from_file(params.model.c_str());

    if (ctx == nullptr) {
        fprintf(stderr, "error: failed to initialize whisper context\n");
        return 3;
    }

    // initialize openvino encoder. This has no effect on whisper.cpp builds that don't have OpenVINO configured.
    whisper_ctx_init_openvino_encoder(ctx, nullptr, params.openvino_encode_device.c_str(), nullptr);

How to generate models and enable OpenVINO for whisper.cpp builds

Here are the instructions for generating the OpenVINO models for use with OpenVINO-enabled builds of whisper.cpp:

• First, setup python virtual env. and install python dependencies. Python 3.10 is recommended.

  Windows:

  cd models
  python -m venv openvino_conv_env
  openvino_conv_env\Scripts\activate
  python -m pip install --upgrade pip
  pip install -r openvino-conversion-requirements.txt
  

  Linux and macOS:

  cd models
  python3 -m venv openvino_conv_env
  source openvino_conv_env/bin/activate
  python -m pip install --upgrade pip
  pip install -r openvino-conversion-requirements.txt
  

• Generate an OpenVINO encoder model. For example, to generate a base.en model, use:

  python convert-whisper-to-openvino.py --model base.en
  

  This will produce ggml-base.en-encoder-openvino.xml/.bin IR model files. It's recommended to relocate these to the same folder as ggml models, as that is the default location that the OpenVINO extension will search at runtime.

• Build whisper.cpp with OpenVINO support:

  Download OpenVINO package from release page. The recommended version to use is 2023.0.0.

  After downloading & extracting package onto your development system, set up required environment by sourcing setupvars script. For example:

  Linux:

  source /path/to/l_openvino_toolkit_ubuntu22_2023.0.0.10926.b4452d56304_x86_64/setupvars.sh

  Windows (cmd):

  C:\Path\To\w_openvino_toolkit_windows_2023.0.0.10926.b4452d56304_x86_64\setupvars.bat
  

  And then build the project using cmake:

  cd build
  cmake -DWHISPER_OPENVINO=1 ..

• Run the examples as usual. For example:

  ./main -m models/ggml-base.en.bin -f samples/jfk.wav
  
  ...
  
  whisper_ctx_init_openvino_encoder: loading OpenVINO model from 'models/ggml-base.en-encoder-openvino.xml'
  whisper_ctx_init_openvino_encoder: first run on a device may take a while ...
  whisper_openvino_init: path_model = models/ggml-base.en-encoder-openvino.xml, device = CPU, cache_dir = models/ggml-base.en-encoder-openvino-cache
  whisper_ctx_init_openvino_encoder: OpenVINO model loaded
  
  system_info: n_threads = 4 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | COREML = 0 | OPENVINO = 1 |
  
  ...

  The first time run on an OpenVINO device is slow, since the OpenVINO framework will compile the IR (Intermediate Representation) model to a device-specific 'blob'. This device-specific blob will get
  cached for the next run.

You can use -oved [DEVICE] argument to main to specify OpenVINO device to offload encoder inference to. For example:

main -m ggml-base.bin -f gb1.wav -oved GPU

Wow - this is quite interesting. First time I hear about OpenVINO - will try to get familiar.
I'll need to look into more details, but overall the implementation looks very nice. Very good PR description

Just to point out -- something that does differ between CoreML and the new OpenVINO integration is how/when support is enabled at runtime. CoreML is enabled within the call to whisper_init_*. For OpenVINO, because we want the ability to specify a device string ("CPU", "GPU", etc.), I exposed a new API that is dedicated to initializing OpenVINO, given a ctx:

Do you think it makes sense to do the same for Core ML so that the implementations follow similar pattern?

@ggerganov, thanks for taking a look!

Do you think it makes sense to do the same for Core ML so that the implementations follow similar pattern?

I think that makes sense, especially if CoreML exposes parameters to control how inference is performed -- but to be honest I know very little about CoreML.

Minor changes - should be good to merge after that

Great stuff 👍

Hi!
in the OpenVino instructions there is the next sentence
cd build
cmake -DWHISPER_OPENVINO=1 ..
where is that "build" dir?

And when I run:
./main -m models/ggml-base.en.bin -f samples/jfk.wav
I don't see the "OPENVINO = 1" or any other info about loading openvino

All the other instructions was executed with success
What is missing?

Distro info
I am running on parrot OS 5.3

Amazing work, and thanks for sharing.

Hi @Nabaralathep,

Looks like I forgot the mkdir build, so it should be:

mkdir build
cd build
cmake -DWHISPER_OPENVINO=1 ..
make

Let me know how it goes.

Hi @RyanMetcalfeInt8,
Thank you very much for your help! You pulled me out of a hole, but... I had some issues that I would like to share to help someone else get out of the hole as well.

1.When I run cmake -DWHISPER_OPENVINO=1 .. the build files are created in the back folder and not in build, maybe because of my cmake version (3.18.4), I solved this with cmake -DWHISPER_OPENVINO=1 .. -B build, all being in the "openvino_conv_env" folder.

2.When I ran make I received an error, it turns out that I had the debian arm version and my computer is x86_64, but when I went to the repository to download the appropriate one, I discovered that all the packages for debian are arm so what?

So this is a dead end, and I'm going to install the pink windows called ubuntu.

In any case, I really appreciate (you don't know how much) your answer, since at least it made me understand the problem, thank you very much and your work is incredible.

Does this implementation of OpenVINO support the GNA in 10th to 14th generation Intel CPUs? Intel advertises it as follows:

Intel® Gaussian & Neural Accelerator is a low-power neural coprocessor for continuous inference at the edge.

When power and performance are critical, the Intel® Gaussian & Neural Accelerator (Intel® GNA) provides power-efficient, always-on support. Intel® GNA is designed to deliver AI speech and audio applications such as neural noise cancellation, while simultaneously freeing up CPU resources for overall system performance and responsiveness.

They also later mention it could be used for tasks such as speech-to-text, and I'm curious if/how well whisper would perform on it.

Setting the OpenVINO device to "gna" just throws an error with assertion failed

whisper_ctx_init_openvino_encoder: loading OpenVINO model from '../../models/ggml-base-encoder-openvino.xml'
whisper_ctx_init_openvino_encoder: first run on a device may take a while ...
whisper_openvino_init: path_model = ../../models/ggml-base-encoder-openvino.xml, device = GNA, cache_dir = ../../models/ggml-base-encoder-openvino-cache
in openvino encoder compile routine: exception: Check 'false' failed at src/inference/src/core.cpp:114:
[ GENERAL_ERROR ]  AssertionFailed: split_sizes.size() > 1

whisper_ctx_init_openvino_encoder: failed to init OpenVINO encoder from '../../models/ggml-base-encoder-openvino.xml'

2.When I ran make I received an error, it turns out that I had the debian arm version and my computer is x86_64, but when I went to the repository to download the appropriate one, I discovered that all the packages for debian are arm so what? !

OpenVINO Ubuntu packages are compatible with Debian OS. You can use OpenVINO archives as well as install via apt and Debian packages.

@ilya-lavrenov -- good suggestions, looks like OpenVINO made some nice improvements for 2023.1+. Did you want to submit a PR with the updates / fixes?