llvm-project

47e1dbeb - [Syntax] Append EOF token to truncated expanded token stream when the parser halts prematurely (#196861)

[Syntax] Append EOF token to truncated expanded token stream when the parser halts prematurely (#196861) Fixes #196244. This PR addresses cases where this assertion is triggered in `TokenCollector::Builder::build()`: https://github.com/llvm/llvm-project/blob/dff356d47cfc4413f78c858dd8339cb1c9fca255/clang/lib/Tooling/Syntax/Tokens.cpp#L715 `TokenCollector` collects the expanded token stream by registering a token watcher callback in the preprocessor. Normally, the preprocessor calls the callback for every token up to and including the `tok::eof` token. However, when the parser hits a hard limit such as exceeding the maximum function scope depth (this is the case covered by #196244) or exceeding the bracket depth limit, it bails out via `Parser::cutOffParsing()`. `cutOffParsing` forces the current token to `eof`, but the token watcher callback is never called for it. The result is a truncated token stream. Fix by checking if `ExpandedTokens` is missing the final `tok::eof`. If so, synthesize one at the location of the last collected token. Includes a test that forces the parser cutoff by exceeding the bracket depth limit. The fixture was extended to allow extra args (the test uses `-fbracket-depth=1`).