transformers
1363fcee - remove the redundant non maintained jieba and use rjieba instead (#40383)

Commit
109 days ago
remove the redundant non maintained jieba and use rjieba instead (#40383) * porting not maintained jieba to rjieba * Fix format * replaced the line with rjieba instead of removing it * cut_all is not included as a parameter. cut_all is a seperate function rjieba * rev * jieba remove installation * Trigger tests * Update tokenization_cpm.py * Update tokenization_cpm_fast.py --------- Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Parents
Loading