llama.cpp
llama : optimize long word tokenization with WPM
#8034
Merged
Overview
Commits
1
Changes
View On GitHub

llama : optimize long word tokenization with WPM #8034

ggerganov merged 1 commit into master from gg/max-token-length
ggerganov
ggerganov llama : optimize long word tokenization with WPM
677bf2e9
ggerganov ggerganov force pushed to 677bf2e9 1 year ago
ggerganov ggerganov merged a927b0f3 into master 1 year ago
ggerganov ggerganov deleted the gg/max-token-length branch 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum