llama.cpp

docs: update function-calling.md w/ template override needed by functionary-small-v3.2

#12214

Merged

Fixes misleading command in doc revealed by #12213 (cc/ @edmcman): bartowski/functionary-small-v3.2-GGUF needs a template override.

Also, now using predownloaded templates under model/templates (used by test_tool_call.py) to simplify commands, and add extra blurb about the python script to get more templates.

Is there any plan to automatically provide the right templates if they are shipped with llama.cpp anyway? I might just apply them in my wrapper but just checking.

Is there any plan to automatically provide the right templates if they are shipped with llama.cpp anyway? I might just apply them in my wrapper but just checking.

I think there are multiple options here:

• (Ideally) Reach out to each maintainer of a GGUF with bad or out of date templates, and have them fix them.
• Detect and explode on "bad" templates (maybe advising on using the GGUF editor to fix the template?)
• Add a simple way to specify which HF repo to pull a template from (-hft / --hf-template flag? add the original HF repo as a key in the GGUF?)
• Bundle a collection of "good" templates and use a mix of predicates on the model metadata to pick the right template (ignoring the template built into the GGUF)

I reckon we could add a flag that switches between the behaviours of the 3 last bullets

cc/ @ngxson @ggerganov WDYT?

If the number of known broken templates are rare (i.e. the case of functionary, it is one of the first model to support function call so it was quite messy), then I don't want to spend too much effort fixing this.

The easiest solution is to simply bundle a collection of good templates as you said on your last point. I believe we will end up with a list of around 10 of them. Newer models should not need this, because they should already have a good built-in template.