Hunyuan tokenizer #26

ngxson merged 13 commits into ngxson:xsn/hunyuan-moe from kooshi:hunyuan
ngxson
kooshi almost working
5e78e887
kooshi skip embed, fix bos
d2195807
kooshi Merge remote-tracking branch 'other/xsn/hunyuan-moe' into hunyuan
616f4c7a
kooshi cleanup
0fd39308
kooshi yarn scaling
b19ecae2
kooshi cleanup
245db159
kooshi failed token fix
8fd547bd
kooshi Merge remote-tracking branch 'other/xsn/hunyuan-moe' into hunyuan
34cc679a
kooshi tokenization working
b20bd263
kooshi Merge remote-tracking branch 'other/xsn/hunyuan-moe' into hunyuan
99d9e946
github-actions github-actions added python
coderabbitai
ngxson
coderabbitai
ngxson
ngxson commented on 2025-06-29
ngxson
ngxson commented on 2025-06-29
ngxson
ngxson commented on 2025-06-29
ngxson
ngxson commented on 2025-06-29
ngxson
ngxson commented on 2025-06-29
ngxson
ngxson commented on 2025-06-29
kooshi cleanup and pr changes
1221d944
kooshi vocab_size sanity check
5471f5ac
kooshi
kooshi ntk alpha generic
46c8b70c
ngxson ngxson merged 443ec9be into xsn/hunyuan-moe 257 days ago
ubergarm

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
Milestone