MTF dataset and packing #293
thomasw21
merged
183 commits into
main
from
thomas/mtf_dataset_and_packing
added train script but with prefix manually declared
1fd80998
made new dataset
fe3f8c27
minor adjustments
31f20877
added capabilities for padding and prefix lm index
6ccacba4
added finetune script
155a8ef6
removed script
ec6c07e5
added adjustments and new dataset
435d65ff
try mlm dataset
b1d7bbde
minor changes
714e5b75
minor addition of import packages
dc154363
minor error fix
e79ac16f
minor error fix
9a90a2e6
samples follow how gpt dataset is loaded
7e79b487
added masked_lm_prob
3453dbdd
added mask id
d382d19a
added mask id
31fbf550
added mask id
f5711327
added mask id
5548a471
added fix
21237f3d
added bos and eos token id
98c46353
no need for sentinal token
e1a75aa6
add aux functions
2fdd7952
add aux functions
154f39c1
add aux functions
3765a81b
add pad_id
2cd41748
changed lm predictions to t5
b592ea33
changed lm predictions to t5
852faca7
changed lm predictions to t5
4333554b
changed lm predictions to t5
be734550
changed lm predictions to t5
163c966e
tokenizer add mask, cls, sep tokens
56de89f5
commit latest changes
ca86fa81
commit latest changes
5011d993
added sentinal tokens
1b15263c
added sentinal tokens
1b165419
added sentinal tokens
0603aac7
added additional_special_tokens
bd061d38
added additional_special_tokens
aff88b91
check t5_input and output
aab47298
check decoder in and decoder out
9448ef48
made into input and output tokens
16ba4aa2
made into input and output tokens
99ca9e80
made into input and output tokens
cdfecad8
made into input and output tokens
e0586881
made into input and output tokens
38ded727
made into input and output tokens
0f68be34
made into input and output tokens
eb84844d
made into input and output tokens
a3af6bf9
made into input and output tokens
6ad61b6d
added eos
9131fdd7
added eos
cb76cd31
test text_token
531ee688
test text_token
a7d11583
test text_token
0008cfb0
test text_token
f1461a83
test text_token
ada0f100
assigned array
298c9b71
assigned array
d2bdff6e
assigned array
4ec8db32
hardcoded sequence length
10a2b6d1
check again
a373a700
show sentinal tokens
bdef71b0
show sentinal tokens
262fd6ce
show sentinal tokens
68a6a936
show sentinal tokens
1c00d4bb
add more special tokens
8b85f113
changed how mlm data is loaded
85d204af
changed how mlm data is loaded
4c842748
changed how mlm data is loaded
084245e5
changed how mlm data is loaded
32af10e8
changed how mlm data is loaded
b6e0e636
added new script
2af2e4b7
added new script
cc5968e5
added new script
cf0b2a0f
try t5 dataset
fc150a05
try t5 dataset
039f90f2
try t5 dataset
7364781e
try t5 dataset
5b1100a4
try t5 dataset
45102a93
try t5 dataset
7b2ebbf7
try t5 dataset
fe8b3dc0
try t5 dataset
f456725e
try t5 dataset
ae73d8cf
try t5 dataset
fae6a0bd
try t5 dataset
81858424
try t5 dataset
9deef493
developing
1e78a4bd
developing
9070929d
developing
56c69de0
developing
d1ca9143
developing
13af6234
developing
dbc555e1
developing
12b209dd
test to see output of get_ltor_masks_and_position_ids
698eff05
test to see output of get_ltor_masks_and_position_ids
dae3cc6c
add new script
5c109c3c
add new script
2fc99951
add new script
ee7af99a
changed settings
b6701a85
changed settings
2283e581
made into input and output tokens
99ca9e80
made into input and output tokens
cdfecad8
made into input and output tokens
e0586881
made into input and output tokens
38ded727
made into input and output tokens
eb84844d
made into input and output tokens
a3af6bf9
made into input and output tokens
6ad61b6d
added eos
9131fdd7
added eos
cb76cd31
test text_token
531ee688
test text_token
a7d11583
test text_token
f1461a83
test text_token
ada0f100
assigned array
298c9b71
assigned array
d2bdff6e
assigned array
4ec8db32
check again
a373a700
show sentinal tokens
bdef71b0
show sentinal tokens
262fd6ce
show sentinal tokens
68a6a936
add more special tokens
8b85f113
changed how mlm data is loaded
85d204af
changed how mlm data is loaded
4c842748
changed how mlm data is loaded
32af10e8
changed how mlm data is loaded
b6e0e636
added new script
2af2e4b7
added new script
cc5968e5
added new script
cf0b2a0f
try t5 dataset
fc150a05
try t5 dataset
7364781e
try t5 dataset
5b1100a4
try t5 dataset
45102a93
try t5 dataset
7b2ebbf7
try t5 dataset
fe8b3dc0
try t5 dataset
f456725e
try t5 dataset
ae73d8cf
try t5 dataset
fae6a0bd
try t5 dataset
81858424
try t5 dataset
9deef493
developing
9070929d
developing
56c69de0
developing
d1ca9143
developing
dbc555e1
test to see output of get_ltor_masks_and_position_ids
698eff05
test to see output of get_ltor_masks_and_position_ids
dae3cc6c
add new script
5c109c3c
add new script
2fc99951
add new script
ee7af99a
changed settings
b6701a85
tidy up
9d00a49f
modifying mlm to reflect original implementation
bde07f08
minor fix
4c0ca2e1
minor fix
30f69248
minor fix
84408ef0
minor fix
ad964c58
minor fix
45899e98
minor fix
2b54cc17
minor fix
ec616272
minor fix
4448d1d3
minor fix
ecd148c7
minor fix
a99f30f0
minor fix
62d3e3e9
minor fix
a1608531
minor fix
fe205f77
minor fix
d39bdaf9
minor fix
2530d3e0
minor fix
5e93c47a
minor fix
ad867998
minor fix
82c8d932
minor fix
ebf3561d
minor fix
811f9755
minor fix
de7dfc83
minor fix
be2af770
minor fix
5e7e18f4
minor fix
24d4f25d
minor fix
5926be1c
minor fix
0f18174c
minor fix
58ce7144
set correct seq len
05470d7c
refined sampling method
51a23f23
refined sampling method
43cb2f04
refined sampling method
901defc8
refined sampling method
3130d7d1
refined sampling method
18eb53d7
refined sampling method
652c545c
first commit, adding non causal mlm dataset
5a49db8e
fixed mlm dataset
81b918c9
fixed mlm dataset
95afc4f0
fixed mlm dataset
c4514d8e
fixed mlm dataset
5cca5af4
fixed mlm dataset
ae958788
minor changes
a03e59f3
removed mlm related scripts
fa1e072d
removed any scipts not related to dataset, revert arguments
e3ce0a76
added sampler and test
87e4055c
added testing data
0ae7661d
adapted test loader
71fb5aea
Update megatron/data/non_causal_mtf_dataset.py
be0cea2d
removed unused files
9daa3766
changed with impossible token
6b9e81a3
added sampler and test
87e4055c
adapted test loader
71fb5aea
Update megatron/data/non_causal_mtf_dataset.py
be0cea2d
removed unused files
9daa3766
enable loading multiple indexed_dataset for each field
7feec27f
minor fix
f84f2935
replace shuffling _build_index_mappings with random.sample of the doc…
cb79f09e
minor changes
e9cf22a3
Cleanup artefacts
acd87cd5
Use seq_length arg
7619f7a6
Update pretrain_t0.py
a7c424e6
WIP
b4e374c4
Cleanup
0d2fdfd6
Add back all configs
126fa34c
Woops
83d24057
Fix tests
c93ed5ce
Rename testing files
528f5d34
Do in-place operations
8bed302d
Do in-place operations
bd2fede1
Woops
8593e425
Fix typo
a1eb558a
thomasw21
commented
on 2022-06-30
Add test that packing is done optimially via greedy algorithm
3bddafa8
Woops
45c94446
Muennighoff
commented
on 2022-06-30
Update tests/test_dataloaders.py
c74dbb7b
Muennighoff
commented
on 2022-06-30
Last segment is either pad or another document that fit perfectly
a337563c
Muennighoff
approved these changes
on 2022-06-30
Resolve conflict
62272326
thomasw21
commented
on 2022-07-02
thomasw21
commented
on 2022-07-02
Login to write a write a comment.
Login via GitHub