NielsRogge/transformers

Pull Requests Commits

Fix TransfoXL constructor vocab_file / pretrained_vocab_file mixin.

mfuntowicz committed 5 years ago

f22083c5

Making sure there is really a bug in unittest

mfuntowicz committed 5 years ago

5b1b723e

mfuntowicz committed 5 years ago

2903c720

Quality and format.

mfuntowicz committed 5 years ago

9fc55e44

Update tokenizers to 0.5.2

mfuntowicz committed 5 years ago

88f709a6

Added some save_pretrained / from_pretrained unittests.

mfuntowicz committed 5 years ago

0cc4c545

TransfoXLTokenizerFast uses a json vocabulary file + warning about incompatibility between Python and Rust

mfuntowicz committed 5 years ago

e0d888d3

Bump tokenizers dependency to 0.5.1

mfuntowicz committed 5 years ago

11269fb6

Remove python quick and dirty fix for clean Rust impl.

mfuntowicz committed 5 years ago

39134cc9

Added save_vocabulary tests.

mfuntowicz committed 5 years ago

d6a24549

Renamed file generate by tokenizers when calling save_pretrained to match python.

mfuntowicz committed 5 years ago

74215920

add explaining example to XLNet LM modeling (#2997)

patrickvonplaten committed 5 years ago

Verified 6a50d501

Add preprocessing step for transfo-xl tokenization to avoid tokenizing words followed by punction to <unk> (#2987)

patrickvonplaten committed 5 years ago

Verified 65d74c49

Add local_files_only parameter to pretrained items (#2930)

Bram Vanroy committed 5 years ago

Verified a143d947

Create README.md

mrm8488 committed 5 years ago

286d1ec7

kwargs are passed to both model and configuration in AutoModels (#2998)

LysandreJik committed 5 years ago

Verified 7984a70e

Testing that batch_encode_plus is the same as encode_plus (#2973)

LysandreJik committed 5 years ago

Verified 21d8b6a3

Add slow generate tests for pretrained lm models (#2909)

patrickvonplaten committed 5 years ago

Verified 17c45c39

Warning on `add_special_tokens` (#2966)

LysandreJik committed 5 years ago

Verified 8194df8e

add_ctags_to_git_ignore (#2984)

patrickvonplaten committed 5 years ago

Verified 38f5fe9e

Delete untested, broken Model2LSTM (#2968)

sshleifer committed 5 years ago

Verified 129f0604

Correct `special_tokens_mask` when `add_special_tokens=False` (#2965)

LysandreJik committed 5 years ago

Verified 0e84559d

Bart: fix layerdrop and cached decoder_input_ids for generation (#2969)

sshleifer committed 5 years ago

Verified 92487a1d

Add standardized get_vocab method to tokenizers

joeddav committed 5 years ago

Verified c36416e5

fix hardcoded path in examples readme

saippuakauppias committed 5 years ago

cafc4dfc

Update modelcard of bert-base-german-cased

tholor committed 5 years ago

34b4b5a9

Update README.md

mrm8488 committed 5 years ago

7df12d7b

Fix max_length not taken into account when using pad_to_max_length on fast tokenizers (#2961)

mfuntowicz committed 5 years ago

Verified cc6775cd

Remove double bias (#2958)

LysandreJik committed 5 years ago

Verified 94ff2d6e

Only use F.gelu for torch >=1.4.0 (#2955)

sshleifer committed 5 years ago

Verified b5b3445c

Older