Sync streaming markdown handling with upstream streamdown (#2311)
* Sync streaming markdown handling with upstream streamdown
Our vendored copies of streamdown's parse-blocks and
parse-incomplete-markdown had drifted behind upstream, missing a number
of streaming correctness and performance fixes.
- Replace vendored parseIncompleteMarkdown with the remend package
(zero-dependency engine extracted from streamdown), picking up:
setext-heading flash guard, single-tilde escaping (20~25 no longer
flashes as strikethrough), comparison-operator escaping in list items,
stripping of half-typed trailing HTML tags, and fixes for emphasis
inside inline code, $$ inside inline code, partial closing $ of block
math, and asterisks inside math blocks.
- Re-port parseBlocks from upstream: tighter footnote detection so
regex character classes like [^\s] in code no longer collapse the
whole message into a single block (which defeated block memoization
while streaming), HTML void element and self-closing tag handling so
<br>/<img>/<div /> no longer swallow all subsequent blocks, nested
same-tag HTML tracking, and skipping $$ math merging after code
blocks (shell $$).
- Render the streamdown:incomplete-link placeholder as an inert anchor
instead of a clickable dead link while a URL is still streaming in.
Verified by porting upstream's test cases into new spec files, a
differential test of parseBlocks against upstream's source across all
streaming prefixes of a 21-document corpus (1154 inputs, 0 mismatches),
and upstream's 388-test remend suite run against the installed package.
https://claude.ai/code/session_01CPQNqQ5F7gio2yBvPM6ihH
* Gate streaming markdown repairs on the loading flag
Apply parseIncompleteMarkdown (remend) only while a message is
streaming, matching upstream streamdown which only runs remend when
mode === "streaming". Completed messages now render their final
markdown unmodified, so e.g. a trailing "Title\n-" setext heading is
no longer rewritten by the streaming flash guard.
Like upstream, the repairs now run on the whole content before block
splitting rather than per block, so block boundaries are computed from
the repaired text and processTokens stays a pure parse.
Addresses the Codex review comment on PR #2311.
https://claude.ai/code/session_01CPQNqQ5F7gio2yBvPM6ihH
---------
Co-authored-by: Claude <noreply@anthropic.com>