transformers

92c5ca9d - Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese (#36121)

Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese (#36121) * Fix potential regex catastrophic backtracking in NougatTokenizerFast The original regex pattern in tokenization_nougat_fast.py was vulnerable to catastrophic backtracking due to greedy quantifiers and nested alternations. This commit replaces it with a more efficient pattern that: 1. Uses explicit character classes instead of dot (.) 2. Handles whitespace more precisely 3. Avoids unnecessary backtracking 4. Supports both lowercase and uppercase roman numerals 5. Maintains the same functionality while being more robust * Try another regex * Trying deepseek's answer * Start with a simplification * Another simplification * Just rewrite the whole function myself * Fix gptneox and gptsan * Simplify the regex even further * Tighten up the price regex a little * Add possessive version of the regex * Fix regex * Much cleaner regexes --------- Co-authored-by: openhands <openhands@all-hands.dev>