llama.cpp
e47e909e - peg-parser: add preserved() for token-aware matching

Commit
116 days ago
peg-parser: add preserved() for token-aware matching Add common_peg_preserved_parser type that matches preserved tokens (like <tool_call>, <escape>) at the token level when token info is available, with graceful fallback to text matching. New features: - common_peg_token_span struct for token boundary info - common_peg_parse_context extended with optional token spans - preserved(literal, token_id) builder method - Token-aware matching: if token_id set and context has tokens, match by token ID; otherwise fall back to text This is the infrastructure for token-aware grammars as described in docs/development/token-aware-grammars.md. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Author
Parents
Loading