transformers
Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese
#36121

Merged

Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese #36121

Rocketknight1 merged 12 commits into main from fix-nougat-regex

Rocketknight1 force pushed from 949dde28 to 75f90bb2 312 days ago

Rocketknight1 marked this pull request as ready for review 312 days ago

Rocketknight1 changed the title ~~Fix potential regex catastrophic backtracking in NougatTokenizerFast~~ Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese 312 days ago

Fix potential regex catastrophic backtracking in NougatTokenizerFast

17f8eaf0

Try another regex

662dc140

Trying deepseek's answer

2d26be35

Start with a simplification

a1a9c962

Another simplification

a6a8c7eb

Just rewrite the whole function myself

f018df0a

Fix gptneox and gptsan

2e4e1e72

Simplify the regex even further

2143d1cf

Tighten up the price regex a little

8c2d71a1

Rocketknight1 force pushed from 404b72d8 to 8c2d71a1 310 days ago

McPatate commented on 2025-02-18

Add possessive version of the regex

80c265c9

Fix regex

7e836fd6

Much cleaner regexes

86a86962

Cyrilvallez approved these changes on 2025-02-21

Rocketknight1 merged 92c5ca9d into main 302 days ago

Rocketknight1 deleted the fix-nougat-regex branch 302 days ago

Reviewers

Cyrilvallez

McPatate

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese #36121 Merged

Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese #36121

transformers
Fix exploitable regexes in Nougat and GPTSan/GPTJNeoXJapanese
#36121

Merged