PR #37 add a test to match author's tokenization

add a test to match author's tokenization #37

SaulLu wants to merge 15 commits into NielsRogge:modeling_markuplm_bis from SaulLu:modeling_markuplm_bis

First draft

64defb25

Make basic test work

eb97daf3

Fix most tokenizer tests

ae8e93b9

More improvements

a062287c

Make more tests pass

48668b23

Fix more tests

a6ee5f68

Fix some code quality

f728bc31

Improve truncation

e3463ca5

Implement feature extractor

8b3b7c97

Improve feature extractor and add tests

aeb7c551

Improve feature extractor tests

10537268

Fix pair_input test partly

bd9e9b8c

Add fast tokenizer

387e52d9

Improve implementation

fd1db5ee

add a test to match author's tokenization

36dcaccd

NielsRogge force-pushed the modeling_markuplm_bis branch from fd1db5ee to f05b8615 3 years ago

NielsRogge force-pushed the modeling_markuplm_bis branch to 049cdbfe 3 years ago

NielsRogge force-pushed the modeling_markuplm_bis branch 3 years ago

NielsRogge force-pushed the modeling_markuplm_bis branch to 738a608e 3 years ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone