PR #7685 Per token attributes

Per token attributes #7685

jaime-m-p merged 10 commits into ggml-org:master from jaime-m-p:per-token-attribs

Add per token attrib enum

cec6a3bd

Using phi-3 for testing 'rstrip'

3ead1b97

bugfix: assertions, wrong special token list

33de2479

Implement 'rstrip' properly

ada961ce

Refactor + add 'jina-v2' for testing 'lstrip'

01c92291

github-actions added testing

github-actions added python

jaime-m-p force pushed from cc55198e to 01c92291 1 year ago

Update phi-3 GGUF file (obsolete since 917dc8c)

8564c198

Update brute force test: testing 'lstrip' and 'rstrip'

83cac0ec

Fix previous commit

54e9f23b

Replace llama_token_type with llama_token_attribs

ac40ff0e

mofosyne added Review Complexity : Medium

ggerganov approved these changes on 2024-06-03

Rename token attributes

18f5fc76

jaime-m-p merged 3b38d486 into master 1 year ago

CISC commented on 2024-06-04

Reviewers

ggerganov

CISC

teleprint-me

Assignees

No one assigned

Labels

testing python Review Complexity : Medium

Milestone

No milestone