NielsRogge/transformers

Pull Requests Commits

Better added_tokens handling

n1t0 committed 5 years ago

e6ec24fa

Code style update

n1t0 committed 5 years ago

599db139

Handle unk_token

n1t0 committed 5 years ago

835b76a4

FastPreTrainedTokenizer => PreTrainedTokenizerFast

n1t0 committed 5 years ago

7ead04ce

Update for changes in tokenizers API

n1t0 committed 5 years ago

1f82a5d9

tokenizers is now a real dependency

n1t0 committed 5 years ago

734d29b0

Add tests for fast tokenizers

n1t0 committed 5 years ago

2818e505

n1t0 committed 5 years ago

31c56f2e

BertTokenizerFast

n1t0 committed 5 years ago

951ae99b

GPT2TokenizerFast

n1t0 committed 5 years ago

041eac2d

FastPreTrainedTokenizer

n1t0 committed 5 years ago

3471ff0d

Merge pull request #2271 from aaugustin/improve-setup-and-requirements

thomwolf committed 5 years ago

Verified 81db12c3

Run the slow tests every Monday morning.

aaugustin committed 5 years ago

10724a81

Remove [--editable] in install instructions.

aaugustin committed 5 years ago

a8d34e53

Merge pull request #2288 from aaugustin/better-handle-optional-imports

aaugustin committed 5 years ago

Verified 072750f4

Use the same pattern as everywhere else.

aaugustin committed 5 years ago

4621ad6f

Reraise ImportError when sentencepiece isn't installed.

aaugustin committed 5 years ago

a31d4a29

Improve exception type.

aaugustin committed 5 years ago

c8b0c1e5

Simplify re-raising exceptions.

aaugustin committed 5 years ago

4c09a960

Remove warning when scikit-learn isn't available.

aaugustin committed 5 years ago

5565dcdd

Run some tests on Python 3.7.

aaugustin committed 5 years ago

8a688182

Remove stray egg-info directory automatically.

aaugustin committed 5 years ago

7a865821

Update contribution instructions.

aaugustin committed 5 years ago

70373a5f

Remove redundant requirements with transformers.

aaugustin committed 5 years ago

c3783399

Remove docs/requirements.txt.

aaugustin committed 5 years ago

d79e9c9a

Remove requirements.txt.

aaugustin committed 5 years ago

d73eb552

Remove requirements-dev.txt.

aaugustin committed 5 years ago

9fcc532d

Include all optional dependencies in extras.

aaugustin committed 5 years ago

76a1417f

Standardize import.

aaugustin committed 5 years ago

9fc8dcb2

Review and update setup.py.

aaugustin committed 5 years ago

f2522869

Older