Megatron-DeepSpeed
MLM adaptation and Multitask Finetuning
#284
Closed
Overview
Commits
150
Changes
View On GitHub

MLM adaptation and Multitask Finetuning #284

lintangsutawika wants to merge 150 commits into bigscience-workshop:main from lintangsutawika:mt0
lintangsutawika
added train script but with prefix manually declared
d2c35fc1
made new dataset
f977b857
minor adjustments
fcfbf176
lintangsutawika added capabilities for padding and prefix lm index
870dfd88
lintangsutawika added finetune script
791bbd02
lintangsutawika removed script
0f44b92d
added adjustments and new dataset
2ff08150
try mlm dataset
f0a79f69
minor changes
eb416c76
minor addition of import packages
c0bc21b6
minor error fix
82e824c0
minor error fix
7bb17ec4
samples follow how gpt dataset is loaded
99297666
added masked_lm_prob
861c41f3
fixed tokenizer abstractions for HF tokenizer
fe951156
added mask id
8ea59430
added mask id
aa0d146a
added mask id
215e8ccb
added mask id
b6eef436
added fix
bfc73a5c
added bos and eos token id
1890f87f
no need for sentinal token
01392a90
add aux functions
923decbe
add aux functions
4611d679
add aux functions
4356de3a
add pad_id
f31c686d
changed lm predictions to t5
a3951e8b
changed lm predictions to t5
97b9a92d
changed lm predictions to t5
fe73a73d
changed lm predictions to t5
6a9cb758
changed lm predictions to t5
469848f3
tokenizer add mask, cls, sep tokens
e68283ff
commit latest changes
476ae94d
commit latest changes
72ff5752
added sentinal tokens
36472913
added sentinal tokens
fcdc9870
added sentinal tokens
d6fbe783
added additional_special_tokens
c44daba0
added additional_special_tokens
a2725d81
check t5_input and output
0e942458
check decoder in and decoder out
b599ab6b
made into input and output tokens
626b0ae3
made into input and output tokens
6008937d
made into input and output tokens
c1524db5
made into input and output tokens
c59c061c
made into input and output tokens
e677e165
made into input and output tokens
9ffaeb9c
made into input and output tokens
d0a6a2fc
made into input and output tokens
47fd987a
made into input and output tokens
4f377e8f
added eos
5c0bf76c
added eos
7c63e4bf
test text_token
871124cc
test text_token
55a593de
test text_token
adb59cae
test text_token
d71afb48
test text_token
7b99bb7c
assigned array
922b09d2
assigned array
469a02db
assigned array
15cb6a0f
hardcoded sequence length
5b0bc176
check again
0671c79d
lintangsutawika show sentinal tokens
6db5c9b0
lintangsutawika show sentinal tokens
8a58007e
lintangsutawika show sentinal tokens
8b0bbc20
lintangsutawika show sentinal tokens
3d1b2560
lintangsutawika add more special tokens
ce00fd9e
lintangsutawika changed how mlm data is loaded
3bcc50c1
lintangsutawika changed how mlm data is loaded
76960f7c
lintangsutawika changed how mlm data is loaded
229d6617
lintangsutawika changed how mlm data is loaded
55e3df75
lintangsutawika changed how mlm data is loaded
05dea6d1
lintangsutawika added new script
661c8bba
lintangsutawika added new script
97d3810f
lintangsutawika added new script
71388ee3
lintangsutawika try t5 dataset
b0f04d5c
lintangsutawika try t5 dataset
cd43a542
lintangsutawika try t5 dataset
e0dc666b
lintangsutawika try t5 dataset
866cee15
lintangsutawika try t5 dataset
0b56a7da
lintangsutawika try t5 dataset
5bb512b9
lintangsutawika try t5 dataset
31d844f1
lintangsutawika try t5 dataset
1d21963b
lintangsutawika try t5 dataset
1429645b
lintangsutawika try t5 dataset
f5341f86
lintangsutawika try t5 dataset
b05b1751
lintangsutawika try t5 dataset
59a6e322
lintangsutawika developing
ab76d492
lintangsutawika developing
0d8dfacf
lintangsutawika developing
e6292245
lintangsutawika developing
efcf50f6
lintangsutawika developing
e5eb6159
lintangsutawika developing
2eee807d
lintangsutawika developing
5840a119
lintangsutawika test to see output of get_ltor_masks_and_position_ids
6d38f735
lintangsutawika test to see output of get_ltor_masks_and_position_ids
430fa6f0
add new script
444314f9
add new script
26c837de
add new script
feb023ca
changed settings
f30b9b16
changed settings
0a9203a5
tidy up
672a8666
changed tokenizer and position embedding
3780e611
modifying mlm to reflect original implementation
2130c315
minor fix
26afe438
minor fix
c1b98168
minor fix
453822f0
minor fix
a62266a6
minor fix
02dda79c
minor fix
80331cb0
minor fix
350227db
minor fix
d0eecd41
minor fix
243cebea
minor fix
da22e0b8
minor fix
083dce75
minor fix
541e9d60
minor fix
86bfc8a4
minor fix
e21a4484
minor fix
f47d6786
minor fix
415b8bce
minor fix
79bd6f8f
minor fix
ba19fdf3
minor fix
d200f4d7
minor fix
102a4614
minor fix
e5304404
minor fix
25680392
minor fix
e6b4120b
minor fix
fd7fe978
minor fix
861fc7b9
minor fix
21c19840
minor fix
14e8d0f6
minor fix
920343f8
minor fix
a68873d8
minor fix
5d439863
set correct seq len
79e8c1a1
refined sampling method
786d252f
refined sampling method
9110520e
refined sampling method
7db34b9e
refined sampling method
d9465158
refined sampling method
bb4e6561
refined sampling method
2e7161d4
first commit, adding non causal mlm dataset
00473e42
fixed mlm dataset
5992776b
fixed mlm dataset
83f5dee9
fixed mlm dataset
3235c2d0
fixed mlm dataset
54499781
fixed mlm dataset
95c98515
Merge branch 'bigscience-workshop:main' into mt0
9ff6172e
minor changes
451318f1
Merge branch 'mt0' of https://github.com/lintangsutawika/Megatron-Dee…
edfaa191
thomasw21
thomasw21 commented on 2022-06-18
thomasw21
thomasw21 commented on 2022-06-18
thomasw21
thomasw21 commented on 2022-06-22
thomasw21
thomasw21 commented on 2022-06-22
Muennighoff
lintangsutawika
thomasw21
lintangsutawika
lintangsutawika lintangsutawika closed this 4 years ago
lintangsutawika lintangsutawika deleted the mt0 branch 4 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum