PR #7245 Unicode codepoint flags for custom regexs

Unicode codepoint flags for custom regexs #7245

jaime-m-p merged 10 commits into ggml-org:master from jaime-m-p:codepoint-flags

mofosyne added Review Complexity : Medium

mofosyne added enhancement

Replace CODEPOINT_TYPE_* with codepoint_flags

e44e6082

Update and bugfix brute force random test

bb205eef

Deterministic brute force random test

707a08d0

Unicode normalization NFD

641944a3

Fix unicode_ranges_nfd

1714e1a7

Minor + style

9bc5d835

Minor + style

a9d8329d

Get rid of BOM

c12db907

Using std::vector instead std::array

6ca6c460

jaime-m-p force pushed from afcbcb55 to 6ca6c460 1 year ago

ggerganov approved these changes on 2024-05-17

Using range_nfd instead of std::tuple

a28dfdc3

jaime-m-p merged b43272af into master 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

enhancement Review Complexity : Medium

Milestone

No milestone