llama.cpp
Correctly identify LF token for GPT-2 style BPE tokenizer
#11496
Merged
Overview
Commits
1
Changes
View On GitHub

Correctly identify LF token for GPT-2 style BPE tokenizer #11496

ggerganov merged 1 commit into ggml-org:master from mgroeber9110:fix-bpe-lf-token
mgroeber9110
mgroeber9110 Correctly identify LF token for GPT-2 style BPE tokenizer
fe8d4df7
ggerganov
ggerganov approved these changes on 2025-01-30
ggerganov ggerganov merged ffd0821c into master 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum