transformers
c0938391 - FIX CIS: big_bird test_fill_mask — locate [MASK] dynamically

Commit
47 days ago
FIX CIS: big_bird test_fill_mask — locate [MASK] dynamically The tokenizer used to fold the space before [MASK] into the special token, keeping it at fixed position 6. Newer tokenizers emit a separate leading '▁' (space) token, shifting [MASK] to position 7 where the model now correctly predicts '<unk>' for the stray space. Drop the space and look up [MASK]'s position dynamically so the test is robust to future tokenizer changes.
Author
Parents
Loading