transformers
f1e8a51f - Preserve spaces in GPT-2 tokenizers (#2778)

Commit
6 years ago
Preserve spaces in GPT-2 tokenizers (#2778) * Preserve spaces in GPT-2 tokenizers Preserves spaces after special tokens in GPT-2 and inhereted (RoBERTa) tokenizers, enabling correct BPE encoding. Automatically inserts a space in front of first token in encode function when adding special tokens. * Add tokenization preprocessing method * Add framework argument to pipeline factory Also fixes pipeline test issue. Each test input now treated as a distinct sequence.
Author
Parents
Loading