Mlm adaptation #287
thomasw21
merged
180 commits into
bigscience-workshop:main
from
lintangsutawika:mlm-adaptation
added train script but with prefix manually declared
d2c35fc1
made new dataset
f977b857
minor adjustments
fcfbf176
added capabilities for padding and prefix lm index
870dfd88
added finetune script
791bbd02
removed script
0f44b92d
added adjustments and new dataset
2ff08150
try mlm dataset
f0a79f69
minor changes
eb416c76
minor addition of import packages
c0bc21b6
minor error fix
82e824c0
minor error fix
7bb17ec4
samples follow how gpt dataset is loaded
99297666
added masked_lm_prob
861c41f3
fixed tokenizer abstractions for HF tokenizer
fe951156
added mask id
8ea59430
added mask id
aa0d146a
added mask id
215e8ccb
added mask id
b6eef436
added fix
bfc73a5c
added bos and eos token id
1890f87f
no need for sentinal token
01392a90
add aux functions
923decbe
add aux functions
4611d679
add aux functions
4356de3a
add pad_id
f31c686d
changed lm predictions to t5
a3951e8b
changed lm predictions to t5
97b9a92d
changed lm predictions to t5
fe73a73d
changed lm predictions to t5
6a9cb758
changed lm predictions to t5
469848f3
tokenizer add mask, cls, sep tokens
e68283ff
commit latest changes
476ae94d
commit latest changes
72ff5752
added sentinal tokens
36472913
added sentinal tokens
fcdc9870
added sentinal tokens
d6fbe783
added additional_special_tokens
c44daba0
added additional_special_tokens
a2725d81
check t5_input and output
0e942458
check decoder in and decoder out
b599ab6b
made into input and output tokens
626b0ae3
made into input and output tokens
6008937d
made into input and output tokens
c1524db5
made into input and output tokens
c59c061c
made into input and output tokens
e677e165
made into input and output tokens
9ffaeb9c
made into input and output tokens
d0a6a2fc
made into input and output tokens
47fd987a
made into input and output tokens
4f377e8f
added eos
5c0bf76c
added eos
7c63e4bf
test text_token
871124cc
test text_token
55a593de
test text_token
adb59cae
test text_token
d71afb48
test text_token
7b99bb7c
assigned array
922b09d2
assigned array
469a02db
assigned array
15cb6a0f
hardcoded sequence length
5b0bc176
check again
0671c79d
show sentinal tokens
6db5c9b0
show sentinal tokens
8a58007e
show sentinal tokens
8b0bbc20
show sentinal tokens
3d1b2560
add more special tokens
ce00fd9e
changed how mlm data is loaded
3bcc50c1
changed how mlm data is loaded
76960f7c
changed how mlm data is loaded
229d6617
changed how mlm data is loaded
55e3df75
changed how mlm data is loaded
05dea6d1
added new script
661c8bba
added new script
97d3810f
added new script
71388ee3
try t5 dataset
b0f04d5c
try t5 dataset
cd43a542
try t5 dataset
e0dc666b
try t5 dataset
866cee15
try t5 dataset
0b56a7da
try t5 dataset
5bb512b9
try t5 dataset
31d844f1
try t5 dataset
1d21963b
try t5 dataset
1429645b
try t5 dataset
f5341f86
try t5 dataset
b05b1751
try t5 dataset
59a6e322
developing
ab76d492
developing
0d8dfacf
developing
e6292245
developing
efcf50f6
developing
e5eb6159
developing
2eee807d
developing
5840a119
test to see output of get_ltor_masks_and_position_ids
6d38f735
test to see output of get_ltor_masks_and_position_ids
430fa6f0
add new script
444314f9
add new script
26c837de
add new script
feb023ca
changed settings
f30b9b16
changed settings
0a9203a5
tidy up
672a8666
changed tokenizer and position embedding
3780e611
modifying mlm to reflect original implementation
2130c315
minor fix
26afe438
minor fix
c1b98168
minor fix
453822f0
minor fix
a62266a6
minor fix
02dda79c
minor fix
80331cb0
minor fix
350227db
minor fix
d0eecd41
minor fix
243cebea
minor fix
da22e0b8
minor fix
083dce75
minor fix
541e9d60
minor fix
86bfc8a4
minor fix
e21a4484
minor fix
f47d6786
minor fix
415b8bce
minor fix
79bd6f8f
minor fix
ba19fdf3
minor fix
d200f4d7
minor fix
102a4614
minor fix
e5304404
minor fix
25680392
minor fix
e6b4120b
minor fix
fd7fe978
minor fix
861fc7b9
minor fix
21c19840
minor fix
14e8d0f6
minor fix
920343f8
minor fix
a68873d8
minor fix
5d439863
set correct seq len
79e8c1a1
refined sampling method
786d252f
refined sampling method
9110520e
refined sampling method
7db34b9e
refined sampling method
d9465158
refined sampling method
bb4e6561
refined sampling method
2e7161d4
first commit, adding non causal mlm dataset
00473e42
fixed mlm dataset
5992776b
fixed mlm dataset
83f5dee9
fixed mlm dataset
3235c2d0
fixed mlm dataset
54499781
fixed mlm dataset
95c98515
Merge branch 'bigscience-workshop:main' into mt0
9ff6172e
minor changes
451318f1
Merge branch 'mt0' of https://github.com/lintangsutawika/Megatron-Dee…
edfaa191
removed multitask finetuning related scripts
56570833
Merge branch 'bigscience-workshop:main' into mlm-adaptation
1cee3458
remove any unrelated to dataset, revert arguments.py
b4b87fcf
revert tokenizer
5e80cc1c
Improve MLM
253e81f0
Woops
1d8a5c09
Remove a bunch of attributes
e6036a06
Fix naming
408f16a0
Woops
ae875529
Use GPTDataset as underlying implementation
e79c9a2e
Fix sep tokens
62ee550c
Change attribute naming
a2e9ba83
GPT Dataset doesn't handle slicing
64334a42
Remove tokenizer
7a872c2e
WIP
b6f02c50
WIP
86680bc6
WIP
4b2d8400
WIP
b935b85b
WIP
9a74d693
WIP
64b1515f
WIP
b2103648
MLM
6398d1d5
Cleanup
e0f7c920
Update megatron/data/mlm_dataset.py
6b929582
TevenLeScao
commented
on 2022-06-26
TevenLeScao
approved these changes
on 2022-06-26
Cleanup + fix off by one issue
faf0b9ec
Missing vocab extra ids
0e3ee159
Woops
92070ce6
Muennighoff
commented
on 2022-06-27
Understanding off by one isse
ea696022
Woops
4dbe4489
Add
8f427900
Muennighoff
approved these changes
on 2022-06-27
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub