main : don't print special tokens with --grammar #6923
mofosyne
merged
15 commits into
ggml-org:master
from
jart:grammar-token
main : don't print special tokens with --grammar
08c5e350
Merge branch 'master' into grammar-token
ea70e280
mofosyne
approved these changes
on 2024-05-18
main: use seperate stream for control characters
bcd24f89
main: use dprintf and add --ctrl-token-no-out and --ctrl-token-fd-out
9f445a79
main: dprintf isn't part of the IEEE POSIX standard. Just use write().
ad4b6097
main: remove --ctrl-token-fd-out in favor for fcntl() based detection
c9ea9df7
common.cpp: accidentally removed --interactive-first
5032f18f
main: only merge stdout and control token if not in conversation or g…
90456a57
main: rejig control token descriptor handling
50048f5b
main: must check pipe status on very top of program
c1e8a6d1
main: renamed --no-special from --ctrl-token-no-out and other refact…
7d52482b
main: refactor ctrl_token_no_out --> no_special
8f76ba54
Merge branch 'master' into grammar-token
d8b373c1
ggerganov
commented
on 2024-05-21
llama: rename llama_token_is_control_token() to llama_token_is_control()
12fcea5d
ggerganov
commented
on 2024-05-21
main: remove special token file descriptor feature (#5)
e75c5ca4
ngxson
approved these changes
on 2024-05-25
Login to write a write a comment.
Login via GitHub