younesbelkada/transformers

Pull Requests Commits

Update test_fill_mask pipeline to not use fast tokenizers.

mfuntowicz committed 6 years ago

a17553b3

Allow to provide tokenizer arguments during pipeline creation.

mfuntowicz committed 6 years ago

c0d38d02

Let's tokenizers handle all the truncation and padding stuff.

mfuntowicz committed 6 years ago

17e1679e

Added use_fast (bool) parameter on AutoTokenizer.from_pretrained().

mfuntowicz committed 6 years ago

888eee72

Add warning message for RobertaTokenizerFast if used for MLM.

mfuntowicz committed 6 years ago

84d40549

_convert_encoding should keep the batch axis tensor if only one sample in the batch.

mfuntowicz committed 6 years ago

3bda68ba

Move max_len_ properties to PreTrainedTokenizerFast and override in specific subclasses.

mfuntowicz committed 6 years ago

2c2bbd15

Expose max_len_ properties on BertTokenizerFast

mfuntowicz committed 6 years ago

2d0063b8

Give correct fixtures path in test_tokenization_fast.py for the CLI.

mfuntowicz committed 6 years ago

cdf33940

Fix tokenization_auto with support for new (python, fast) mapping schema.

mfuntowicz committed 6 years ago

d6af63db

mfuntowicz committed 6 years ago

66cd67eb

Bump tokenizers dependency to 0.3.0

mfuntowicz committed 6 years ago

eb518cbe

Bind all the encode_plus parameter to the forwarded batch_encode_plus call.

mfuntowicz committed 6 years ago

5631450a

batch_encode_plus now dispatchs to encode if there is only one input element.

mfuntowicz committed 6 years ago

fa25b950

encode_plus dispatchs to batch_encode_plus

mfuntowicz committed 6 years ago

1d74882f

Introduce tokenize() method on fast tokenizers.

mfuntowicz committed 6 years ago

c44b8a33

Use "fast" suffix instead of "ru" for rust tokenizers implementations.

mfuntowicz committed 6 years ago

09063fe6

mfuntowicz committed 6 years ago

cfedc333

Rename LookupTable to WordLevel to match Rust side.

mfuntowicz committed 6 years ago

be1a33d8

mfuntowicz committed 6 years ago

6ac32bca

Added tokenizers as top-level modules.

mfuntowicz committed 6 years ago

2cd49978

Updating tests accordingly.

mfuntowicz committed 6 years ago

ddad200b

Binding TransformerXL tokenizers to Rust.

mfuntowicz committed 6 years ago

71d1adbb

Removed fast implementation of CTRL.

mfuntowicz committed 6 years ago

923b15f1

Matching Roberta tokenization !

mfuntowicz committed 6 years ago

0f091e2a

Expose add_prefix_space as constructor parameter for GPT2

mfuntowicz committed 6 years ago

eb418f09

Matching GPT2 on tokenizers

mfuntowicz committed 6 years ago

2ae31e71

Matching OpenAI GPT tokenization !

mfuntowicz committed 6 years ago

8c738fbb

Added matching tests

mfuntowicz committed 6 years ago

99796888

Bumped tokenizers version requirements to latest 0.2.1

mfuntowicz committed 6 years ago

5495c7d4

Older