transformers
remove the redundant non maintained jieba to rjieba
#40383

Merged

remove the redundant non maintained jieba to rjieba #40383

ydshieh merged 10 commits into huggingface:main from divyanshsinghvi:porting_jieba_dependency_to_rjieba

divyanshsinghvi force pushed from 48158ef1 to 6a85fa75 120 days ago

porting not maintained jieba to rjieba

49f8c574

Fix format

7d10da8f

replaced the line with rjieba instead of removing it

1e3b019a

cut_all is not included as a parameter. cut_all is a seperate functio…

a93d6900

rev

6297bfac

divyanshsinghvi force pushed from 45c21cd8 to 6297bfac 117 days ago

Ed-uardo commented on 2025-08-26

divyanshsinghvi requested a review from

Ed-uardo 116 days ago

jieba remove installation

5dacc797

divyanshsinghvi force pushed from 5cc73b44 to 5dacc797 115 days ago

Trigger tests

f4c6ec33

Ed-uardo approved these changes on 2025-08-28

Merge branch 'main' into porting_jieba_dependency_to_rjieba

83a43fdf

ydshieh commented on 2025-08-28

ydshieh approved these changes on 2025-08-28

Update tokenization_cpm.py

ac7dbc41

Update tokenization_cpm_fast.py

709c9c5f

divyanshsinghvi requested a review from

ydshieh 114 days ago

ydshieh approved these changes on 2025-08-28

ydshieh merged 1363fcee into main 112 days ago

Reviewers

ydshieh

Ed-uardo

Assignees

No one assigned

Labels

None yet

Milestone

No milestone