PR #12532 llama-vocab : add SuperBPE pre-tokenizer

llama-vocab : add SuperBPE pre-tokenizer #12532

ngxson merged 1 commit into master from compilade/superbpe

llama-vocab : add SuperBPE pre-tokenizer

35c2f8b9

compilade requested a review from

ngxson 336 days ago

github-actions added python

compilade added model

ngxson approved these changes on 2025-03-23

ngxson merged 00d53800 into master 335 days ago

Reviewers

ngxson

Assignees

No one assigned

Labels

model python

Milestone

No milestone