transformers
1656d90b - Fix TAPAS tokenizer crash on pandas 3.x string-dtype tables (#45916)

Commit
107 days ago
Fix TAPAS tokenizer crash on pandas 3.x string-dtype tables (#45916) * Fix TAPAS tokenizer crash on pandas 3.x string-dtype tables `add_numeric_table_values` overwrites every cell of the input DataFrame with a `Cell` dataclass instance. In pandas 3.x, string columns default to the pyarrow-backed `StringDtype`, whose `_maybe_convert_setitem_value` calls `len()` on the assignee to distinguish scalars from sequences. The `Cell` dataclass has no `__len__`, so the write raises `TypeError`, crashing `table-question-answering` pipelines on pandas 3.x. Cast the (already-copied) table to `object` dtype before the loop so arbitrary Python objects are accepted again. Behavior on pandas 2.x is unchanged. Fixes #45901. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Fix the behaviour and add a proper end-to-end test --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
Parents
Loading