feat: add changes to handle jina v2 base code #7596
📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 527 iterations 🚀
Expand details for performance related PR only
- Concurrent users: 8, duration: 10m
- HTTP request : avg=8891.02ms p(95)=21954.48ms fails=, finish reason: stop=474 truncated=53
- Prompt processing (pp): avg=104.81tk/s p(95)=444.45tk/s
- Token generation (tg): avg=45.37tk/s p(95)=46.03tk/s
- ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=feat-jina-v2-base-code commit=4c4d877d23dd27fc7e323b4a2623db825e8bd29f
More
---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 527 iterations"
y-axis "llamacpp:prompt_tokens_seconds"
x-axis "llamacpp:prompt_tokens_seconds" 1717659739 --> 1717660371
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 493.19, 493.19, 493.19, 493.19, 493.19, 857.68, 857.68, 857.68, 857.68, 857.68, 709.43, 709.43, 709.43, 709.43, 709.43, 733.5, 733.5, 733.5, 733.5, 733.5, 767.28, 767.28, 767.28, 767.28, 767.28, 784.63, 784.63, 784.63, 784.63, 784.63, 791.91, 791.91, 791.91, 791.91, 791.91, 802.36, 802.36, 802.36, 802.36, 802.36, 800.36, 800.36, 800.36, 800.36, 800.36, 819.58, 819.58, 819.58, 819.58, 819.58, 843.83, 843.83, 843.83, 843.83, 843.83, 841.72, 841.72, 841.72, 841.72, 841.72, 845.53, 845.53, 845.53, 845.53, 845.53, 851.97, 851.97, 851.97, 851.97, 851.97, 844.57, 844.57, 844.57, 844.57, 844.57, 842.59, 842.59, 842.59, 842.59, 842.59, 849.22, 849.22, 849.22, 849.22, 849.22, 846.64, 846.64, 846.64, 846.64, 846.64, 845.18, 845.18, 845.18, 845.18, 845.18, 840.62, 840.62, 840.62, 840.62, 840.62, 837.89, 837.89, 837.89, 837.89, 837.89, 845.37, 845.37, 845.37, 845.37, 845.37, 848.1, 848.1, 848.1, 848.1, 848.1, 859.19, 859.19, 859.19, 859.19, 859.19, 825.7, 825.7, 825.7, 825.7, 825.7, 828.95, 828.95, 828.95, 828.95, 828.95, 830.58, 830.58, 830.58, 830.58, 830.58, 843.53, 843.53, 843.53, 843.53, 843.53, 842.45, 842.45, 842.45, 842.45, 842.45, 842.1, 842.1, 842.1, 842.1, 842.1, 846.88, 846.88, 846.88, 846.88, 846.88, 848.49, 848.49, 848.49, 848.49, 848.49, 845.74, 845.74, 845.74, 845.74, 845.74, 849.84, 849.84, 849.84, 849.84, 849.84, 861.69, 861.69, 861.69, 861.69, 861.69, 864.26, 864.26, 864.26, 864.26, 864.26, 859.05, 859.05, 859.05, 859.05, 859.05, 858.1, 858.1, 858.1, 858.1, 858.1, 855.1, 855.1, 855.1, 855.1, 855.1, 855.09, 855.09, 855.09, 855.09, 855.09, 859.42, 859.42, 859.42, 859.42, 859.42, 860.42, 860.42, 860.42, 860.42, 860.42, 864.41, 864.41, 864.41, 864.41, 864.41, 856.43, 856.43, 856.43, 856.43, 856.43, 845.97, 845.97, 845.97, 845.97, 845.97, 844.92, 844.92, 844.92, 844.92, 844.92, 843.12, 843.12, 843.12, 843.12, 843.12, 845.46, 845.46, 845.46, 845.46, 845.46, 846.38, 846.38, 846.38, 846.38, 846.38, 845.88, 845.88, 845.88, 845.88, 845.88, 847.71, 847.71, 847.71, 847.71, 847.71, 850.33, 850.33, 850.33, 850.33, 850.33, 853.11, 853.11, 853.11, 853.11, 853.11, 856.25, 856.25, 856.25, 856.25, 856.25, 854.69, 854.69, 854.69, 854.69, 854.69, 859.26, 859.26, 859.26, 859.26, 859.26, 860.76, 860.76, 860.76, 860.76, 860.76, 862.36, 862.36, 862.36, 862.36, 862.36, 861.9, 861.9, 861.9, 861.9, 861.9, 862.47, 862.47, 862.47, 862.47, 862.47, 862.06, 862.06]
More
---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 527 iterations"
y-axis "llamacpp:predicted_tokens_seconds"
x-axis "llamacpp:predicted_tokens_seconds" 1717659739 --> 1717660371
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.24, 36.24, 36.24, 36.24, 36.24, 38.3, 38.3, 38.3, 38.3, 38.3, 28.79, 28.79, 28.79, 28.79, 28.79, 29.62, 29.62, 29.62, 29.62, 29.62, 30.21, 30.21, 30.21, 30.21, 30.21, 30.95, 30.95, 30.95, 30.95, 30.95, 32.21, 32.21, 32.21, 32.21, 32.21, 32.32, 32.32, 32.32, 32.32, 32.32, 33.43, 33.43, 33.43, 33.43, 33.43, 33.73, 33.73, 33.73, 33.73, 33.73, 33.57, 33.57, 33.57, 33.57, 33.57, 33.77, 33.77, 33.77, 33.77, 33.77, 33.22, 33.22, 33.22, 33.22, 33.22, 32.28, 32.28, 32.28, 32.28, 32.28, 31.88, 31.88, 31.88, 31.88, 31.88, 31.02, 31.02, 31.02, 31.02, 31.02, 29.89, 29.89, 29.89, 29.89, 29.89, 30.23, 30.23, 30.23, 30.23, 30.23, 30.23, 30.23, 30.23, 30.23, 30.23, 29.83, 29.83, 29.83, 29.83, 29.83, 29.3, 29.3, 29.3, 29.3, 29.3, 29.3, 29.3, 29.3, 29.3, 29.3, 29.46, 29.46, 29.46, 29.46, 29.46, 29.66, 29.66, 29.66, 29.66, 29.66, 29.48, 29.48, 29.48, 29.48, 29.48, 29.69, 29.69, 29.69, 29.69, 29.69, 29.89, 29.89, 29.89, 29.89, 29.89, 30.0, 30.0, 30.0, 30.0, 30.0, 29.76, 29.76, 29.76, 29.76, 29.76, 29.92, 29.92, 29.92, 29.92, 29.92, 30.18, 30.18, 30.18, 30.18, 30.18, 30.23, 30.23, 30.23, 30.23, 30.23, 30.39, 30.39, 30.39, 30.39, 30.39, 30.52, 30.52, 30.52, 30.52, 30.52, 30.61, 30.61, 30.61, 30.61, 30.61, 30.47, 30.47, 30.47, 30.47, 30.47, 30.39, 30.39, 30.39, 30.39, 30.39, 30.03, 30.03, 30.03, 30.03, 30.03, 29.67, 29.67, 29.67, 29.67, 29.67, 29.78, 29.78, 29.78, 29.78, 29.78, 29.8, 29.8, 29.8, 29.8, 29.8, 29.96, 29.96, 29.96, 29.96, 29.96, 30.07, 30.07, 30.07, 30.07, 30.07, 29.92, 29.92, 29.92, 29.92, 29.92, 29.68, 29.68, 29.68, 29.68, 29.68, 29.53, 29.53, 29.53, 29.53, 29.53, 28.64, 28.64, 28.64, 28.64, 28.64, 28.28, 28.28, 28.28, 28.28, 28.28, 28.23, 28.23, 28.23, 28.23, 28.23, 28.23, 28.23, 28.23, 28.23, 28.23, 28.28, 28.28, 28.28, 28.28, 28.28, 28.33, 28.33, 28.33, 28.33, 28.33, 28.41, 28.41, 28.41, 28.41, 28.41, 28.45, 28.45, 28.45, 28.45, 28.45, 28.33, 28.33, 28.33, 28.33, 28.33, 28.32, 28.32, 28.32, 28.32, 28.32, 28.29, 28.29, 28.29, 28.29, 28.29, 28.32, 28.32, 28.32, 28.32, 28.32, 28.55, 28.55, 28.55, 28.55, 28.55, 28.67, 28.67, 28.67, 28.67, 28.67, 28.75, 28.75]
Details
More
---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 527 iterations"
y-axis "llamacpp:kv_cache_usage_ratio"
x-axis "llamacpp:kv_cache_usage_ratio" 1717659739 --> 1717660371
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.13, 0.13, 0.13, 0.13, 0.13, 0.43, 0.43, 0.43, 0.43, 0.43, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.14, 0.14, 0.14, 0.14, 0.14, 0.09, 0.09, 0.09, 0.09, 0.09, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.3, 0.3, 0.3, 0.3, 0.3, 0.29, 0.29, 0.29, 0.29, 0.29, 0.43, 0.43, 0.43, 0.43, 0.43, 0.43, 0.43, 0.43, 0.43, 0.43, 0.25, 0.25, 0.25, 0.25, 0.25, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.33, 0.33, 0.33, 0.33, 0.33, 0.35, 0.35, 0.35, 0.35, 0.35, 0.1, 0.1, 0.1, 0.1, 0.1, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1, 0.1, 0.1, 0.1, 0.1, 0.11, 0.11, 0.11, 0.11, 0.11, 0.16, 0.16, 0.16, 0.16, 0.16, 0.31, 0.31, 0.31, 0.31, 0.31, 0.22, 0.22, 0.22, 0.22, 0.22, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.18, 0.18, 0.18, 0.18, 0.18, 0.11, 0.11, 0.11, 0.11, 0.11, 0.18, 0.18, 0.18, 0.18, 0.18, 0.31, 0.31, 0.31, 0.31, 0.31, 0.32, 0.32, 0.32, 0.32, 0.32, 0.31, 0.31, 0.31, 0.31, 0.31, 0.38, 0.38, 0.38, 0.38, 0.38, 0.13, 0.13, 0.13, 0.13, 0.13, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.27, 0.27, 0.27, 0.27, 0.27, 0.57, 0.57, 0.57, 0.57, 0.57, 0.58, 0.58, 0.58, 0.58, 0.58, 0.66, 0.66, 0.66, 0.66, 0.66, 0.38, 0.38, 0.38, 0.38, 0.38, 0.29, 0.29, 0.29, 0.29, 0.29, 0.22, 0.22, 0.22, 0.22, 0.22, 0.24, 0.24, 0.24, 0.24, 0.24, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.33, 0.33, 0.33, 0.33, 0.33, 0.24, 0.24, 0.24, 0.24, 0.24, 0.2, 0.2, 0.2, 0.2, 0.2, 0.11, 0.11, 0.11, 0.11, 0.11, 0.14, 0.14, 0.14, 0.14, 0.14, 0.11, 0.11, 0.11, 0.11, 0.11, 0.12, 0.12, 0.12, 0.12, 0.12, 0.21, 0.21]
More
---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 527 iterations"
y-axis "llamacpp:requests_processing"
x-axis "llamacpp:requests_processing" 1717659739 --> 1717660371
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0]
It's how the tokens are handled in llama.cpp. I'm in the middle of figuring out how tokenizers operates under the hood and seeing if there's a way to create a bridge between the two. Actually, your input would be invaluable (#7379). Or if you know someone that's better suited and has a deeper understanding of tokenizers (e.g. BPE/WPM) in general. I'm interested in Jina because the english version uses WPM. The spanish and dutch versions use BPE. I'm more focused on Llama-2 and Llama-3 for BPE.
Aside: I have no idea how the CD/CI is setup here. I have some experience with Jenkins, but all of this is outside of the scope of what I'm focused on. Also, I'm just a contributor. I just chime in when I think I might have something of value to add.
It's how the tokens are handled in llama.cpp. I'm in the middle of figuring out how
tokenizersoperates under the hood and seeing if there's a way to create a bridge between the two. Actually, your input would be invaluable (#7379). Or if you know someone that's better suited and has a deeper understanding of tokenizers (e.g. BPE/WPM) in general. I'm interested in Jina because the english version uses WPM. The spanish and dutch versions use BPE. I'm more focused on Llama-2 and Llama-3 for BPE.Aside: I have no idea how the CD/CI is setup here. I have some experience with Jenkins, but all of this is outside of the scope of what I'm focused on. Also, I'm just a contributor. I just chime in when I think I might have something of value to add.
Hey @teleprint-me ,
To be honest, I found it quite hard to work with tokenizer logic here, but I do not quite understand what you aim to achieve in #7379. If you want we can jump in a call to discuss and make this process more agile.
could you also guide me on how to fix the CI problems?
Rebase on latest master and the CI should work
could you also guide me on how to fix the CI problems?
Rebase on latest
masterand the CI should work
I will, thanks
So ):\ tit should not be matched. Is there any logic in the code that eliminates these patterns \ from the vocab?
Hm, not sure why this happens. We don't escape strings in the vocab - only in the prompt input:
So ):\ tit should not be matched. Is there any logic in the code that eliminates these patterns \ from the vocab?
Hm, not sure why this happens. We don't escape strings in the vocab - only in the prompt input:
I will try to investigate this
I am also trying to see if I can add support for chinese model and I manage to get it to work for English, but not for the Chinese characters. Is there a supported model in Chinese? So I can see if I can inspire on which tokenizers they use, etc ...?
I believe the most recent model that we added and also supports Chinese is https://huggingface.co/deepseek-ai/DeepSeek-V2. See if @fairydreaming's PR could be of any help: #7519
Hey @ggerganov ,
I am starting to think that it is not a problem of the tokenizer.
Here is my observation.
I am tryng to run this code to check how the embedding behaves:
gdb --args ../build/bin/embedding -m ./jina-embeddings-v2-base-code.gguf --threads 1 --verbose-prompt -p "for idx, x in enumerate(xs):\n print(idx, x)"and this is what gdb is telling me:
(gdb) run
Starting program: /home/joan/workspace/ollama/llm/llama.cpp/build/bin/embedding -m ./jina-embeddings-v2-base-code.gguf --threads 1 --verbose-prompt -p for\ idx,\ x\ in\ enumerate\(xs\):\\n\ \ \ \ print\(idx,\ x\)
Look at all the \ that have been added. This seems to be the reason why I get different tokenization, in Python If I add an extra \ before \\n I get the same encoding.
I am not sure if it is a problem of how the standard input is encoded or something? Do you happen to have any clue about this?
if I hardcode this sentence and avoid the split lines.
params.prompt = "for idx, x in enumerate(xs):\n print(idx, x)";
I get the same behavior as in Python
I see, does adding -e to the command-line argument fix the issue?
../build/bin/embedding -m ./jina-embeddings-v2-base-code.gguf --threads 1 --verbose-prompt -e -p "for idx, x in enumerate(xs):\n print(idx, x)"
I see, does adding
-eto the command-line argument fix the issue?../build/bin/embedding -m ./jina-embeddings-v2-base-code.gguf --threads 1 --verbose-prompt -e -p "for idx, x in enumerate(xs):\n print(idx, x)"
Oh, it does!
I see, does adding
-eto the command-line argument fix the issue?../build/bin/embedding -m ./jina-embeddings-v2-base-code.gguf --threads 1 --verbose-prompt -e -p "for idx, x in enumerate(xs):\n print(idx, x)"
how then can we be sure this behavior is available in the server? I see this escape option only available in the example itself.
I believe server already escapes these through the JSON parsing library. Btw all examples now escape by default since #7675, so no need to even add -e explicitly
Hey @ggerganov,
Is there something from my code that may have caused this CI to fail?
Probably just a fluke, will restart the workflows now
@ggerganov I tested the behavior in server and works, I consider this is ready to be reviewed.
style : minor
Login to write a write a comment.
PR to allow using
jinaai/jina-embeddings-v2-base-codewithllama.cpp. It has an extra normalization layer compared to other models of theJinaV2family and this is why it is considered independently.