nvda

7c99c661 - Add ICU word segmentation backend for browse mode word navigation (#20379)

Add ICU word segmentation backend for browse mode word navigation (#20379) Closes #20343 ### Summary of the issue: NVDA's word navigation in browse mode uses Windows Uniscribe (`ScriptBreak`), which has no dictionary-based segmentation for scripts that don't separate words with spaces. As a result, word navigation steps through Japanese text (and other complex scripts) one character at a time instead of moving by linguistic word. Multi-character emoji (ZWJ sequences) are likewise split. ### Description of user facing changes: - A new "Windows Unicode (ICU)" option is added to the Word Segmentation Standard setting in the "Document Navigation" panel. - Under "Auto", word navigation now prefers ICU over the legacy Windows (Uniscribe) segmentation wherever ICU is available. Chinese word segmentation (cppjieba) continues to take precedence for Chinese text. - The existing "Standard" option is relabelled "Windows (legacy)" to make clear it is the older Uniscribe path. - Word navigation by word now works correctly for Japanese, Khmer and other complex scripts, and for multi-character emoji sequences, where the legacy segmentation previously fell back to character-level boundaries. ### Description of developer facing changes: - New `WordSegFlag.ICU` flag and `WordNavigationUnitFlag.ICU` feature-flag enum value. - New `IcuWordSegmentationStrategy` in the `_wordSeg` strategy framework, backed by new low-level modules `winBindings/icu.py` (ctypes bindings to the Windows built-in ICU `ubrk_*` BreakIterator API) and `textUtils/icu.py` (`calculateWordOffsets`). Word boundaries follow Unicode Standard Annex #29 plus automatic dictionary-based segmentation selected by the script of the text. - `WordSegmenter._chooseStrategy` reworked into an explicit fallback chain: Chinese (cppjieba) → ICU → Uniscribe. ICU is selected for the `AUTO` and `ICU` flags and as the fallback when cppjieba is unavailable; Uniscribe remains the final fallback and the only strategy for the explicit `UNISCRIBE` flag (it stays pinned where strictly required, e.g. `EditTextInfo`). ### Description of development approach: ICU was integrated into the existing `_wordSeg` strategy framework introduced by the cppjieba PR (#20183), so that strategy selection lives in one place. The ICU layer is offset-only: `IcuWordSegmentationStrategy.segmentedText` returns the text unchanged (no braille separator insertion), so braille output is unaffected. Offsets are converted to/from UTF-16 for ICU. The ICU primitives use the root locale unconditionally because word boundaries are script-driven, not locale-driven. Trailing whitespace is attached to the preceding word to match NVDA's existing Uniscribe behaviour. This PR scopes ICU to **word** segmentation only. ICU integration can be broadened in follow-ups to also drive **character**, **line** and **sentence** boundary detection, which would benefit from the same UAX#29 handling (e.g. grapheme clusters for character navigation). ### Testing strategy: - Unit tests for the ICU word offset calculation (`test_wordSegIcu.py`) covering UAX#29 boundaries, dictionary-segmented scripts, whitespace attachment, surrogate pairs and offset round-tripping. - A backend comparison test (`test_textUtils_backendComparison.py`) asserting ICU vs Uniscribe divergence on Japanese/Khmer and on a multi-person emoji ZWJ sequence with skin-tone modifiers, plus parity on common cases. - Manual testing of word navigation in browse mode across Japanese, Khmer, emoji sequences and Chinese (cppjieba precedence preserved). ### Known issues with pull request: - ICU requires Windows 10 version 1703 (Creators Update) or later; on older systems NVDA falls back to Uniscribe. - ICU coalesces a run of identical whitespace into one segment but splits mixed whitespace (space + tab) into separate segments. Not special-cased — legacy Uniscribe behaviour for mixed runs is itself inconsistent. - ICU splits some tokens that Uniscribe keeps whole (e.g. `well-known` → `well`/`-`/`known`, `a@b.com`). Tradeoff of UAX#29 default rules. - ICU treats trailing punctuation as a separate word, so word navigation stops on it independently (e.g. `logo.` → `logo` then `.`), whereas Uniscribe kept the punctuation attached to the preceding word. This matches the word-navigation behaviour of modern Windows edit controls such as the Start menu search field.