transformers
fda36053 - Fix run_glue.py label ids becoming strings with pandas>=3 (#49326)

Commit
11 days ago
Fix run_glue.py label ids becoming strings with pandas>=3 (#49326) With pandas>=3, `datasets` loads the text columns of CSV files as `large_string`. When `run_glue.py` maps the labels to ids, `datasets.map` first tries the column's original type, and its guard against casting ints to strings only covers `string`, not `large_string`. The ids then come back as `'0'`/`'1'` and the data collator fails with `ValueError: too many dimensions 'str'` (seen in `test_run_glue` on Python 3.11, where pandas 3 gets installed). Cast a `large_string` label column to `string` after loading, so the ids are inferred as ints again. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Author
Parents
Loading