Remove token-aware parsing infrastructure
This feature was added early in the branch but proved not useful
in practice. Removing to simplify the PEG parser codebase.
Changes:
- Replace p.token() with p.literal() in all chat-parsers
- Replace p.token_tag() with p.atomic_tag(p.literal()) in chat-parsers
- Remove common_peg_token_parser struct from peg-parser.h
- Remove common_peg_until_token_parser struct from peg-parser.h
- Remove token_ids and tokens fields from parse context
- Remove token parser execution, serialization, and GBNF generation
- Remove token parser tests from test-basic.cpp and test-gbnf-generation.cpp
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
