Megatron-DeepSpeed
Prefix lm
#52
Merged
Overview
Commits
98
Changes
View On GitHub

Prefix lm #52

thomasw21 merged 98 commits into bigscience-workshop:main from thomasw21:prefix_lm
thomasw21
mpatwary ICT zeroshot evaluation code
8b04e0e4
mpatwary made more generic, aligned with other tasks
661553f6
mpatwary Fixed based on review recoemmendation
43c9137b
mpatwary fixed another issue
40565390
mpatwary Merge branch 'main' into main_retriver_merge_ict_eval
a5acbf53
mpatwary implementing DPR
10ff0607
mpatwary Merge branch 'main' into main_retriver_merge_dpr
cdde4338
mpatwary implementation dpr
06076c7a
Merge branch 'main' into main_retriver_merge_dpr
957d1c9a
adding dpr code
b9fcb7b4
removed commnets
80047314
removed commnets
f415dc85
removed commnets
a8d172b3
DPR evaluation debugging
220637f9
DPR ongoing
d2d5086e
DPR finetune and evaluation
6d03d7af
fixing model evaluation of retriver
f9267205
added pre ad post process
54093417
added pre ad post process
7e335e15
evaluation works!
f64977fd
debugging DPR
dca47cfb
stas00 fix copy-n-paste error
3f755374
devrimcavusoglu Typo fix in readme
07ca952e
stas00 t5 fixes
2dae74b6
mpatwary Merge branch 'main' into main_retriver_merge_dpr
4a09bb3c
mpatwary before cleaning the comments
7a0710ec
kvareddy vit pipeline fixes
ccae9dbd
mpatwary cleaning the code
2eaf6c79
mpatwary additional cleaning
25293807
mpatwary renaming the folders
8e44d619
jaredcasper Add temporary assert to finetuning until it can be fixed.
113c6362
mpatwary Fixed issues with ICT pretraining
7577931b
mpatwary updated the evaluation script for retriver
dfb6a9b3
mpatwary updated the evaluation script for retriver
f21a6628
mpatwary updated the evaluation script for retriver
a41e4781
mpatwary updated the evaluation script for retriver
825375cf
shoeybi Merge branch 'finetune_assert' into 'main'
217f54b3
mpatwary added exit interval for finetuning
d078e54a
mpatwary updating the scripts
63121a9e
mpatwary updating no load rng
fda81a21
jaredcasper Merge branch 'vit_pipeline_fixes' into 'main'
01fc0833
jaredcasper Merge branch 'main_retriver_merge_dpr' into 'main'
83c4d95a
mpatwary updating script
c7c65bbb
mpatwary Merge branch 'main' into main_retriver_merge_dpr
84eb016c
deepakn94 Update T5 scripts
3dadd16d
mpatwary resolved hang issue
04c79f30
mpatwary fixed the tensor size miss-mass issue
ebfbfcec
mpatwary fixed the evaluation hangs
e46f3260
mpatwary Adding readme
a983cab3
mpatwary Adding readme
d562d7b5
mpatwary Adding readme
1095d7e6
mpatwary Adding readme
bab5cc4e
mpatwary Adding readme
8661ca26
mpatwary Adding readme
293554aa
mpatwary Adding readme
e287bf00
mpatwary Adding readme
c45109ed
jaredcasper Clean up README.md a bit
473127f9
mpatwary addressed comments
28450473
mpatwary Merge branch 'main_retriver_merge_dpr' of ssh://gitlab-master.nvidia.…
98113c69
jaredcasper Merge branch 'main_retriver_merge_dpr' into 'main'
598d7ee2
jaredcasper Merge branch 't5_scripts' into 'main'
2be1e510
mpatwary updated readme
9d350c9c
mpatwary updated readme
baf2e2a4
mpatwary updated readme
32da2e78
mpatwary updated readme
4c92ca82
jaredcasper Merge branch 'main_retriver_merge_dpr' into 'main'
82b69e86
jaredcasper Merge branch 't5' of https://github.com/stas00/Megatron-LM into githu…
7898c9a4
jaredcasper Merge branch 'typo-fix' of https://github.com/devrimcavusoglu/Megatro…
e1318f0e
jaredcasper Merge branch 'patch-1' of https://github.com/stas00/Megatron-LM into …
4a35d50a
jaredcasper Merge branch 'github-pr' into 'main'
90e0a0dd
thomasw21 Basic handling of prefix lm by updating the mask
b9fbe3db
thomasw21 Add prefix option to gpt temporarily and prevent it to use custom kernel
f69a0023
thomasw21 Add argument for prefix lm, in order to configure masking strategy
c63eb38f
thomasw21 Woops
bc5313bf
thomasw21
thomasw21 commented on 2021-08-05
thomasw21 loss_on_targets_only flag, assert that current prefix implementation …
60a5884d
thomasw21 Format
fa134edf
huu4ontocord
huu4ontocord commented on 2021-08-05
huu4ontocord
huu4ontocord commented on 2021-08-05
thomasw21 Reverse renaming
3b678fe0
thomasw21 Allow prefix on partial document at the end
fe5f9b6a
thomasw21 WIP: add prefix per row feature
a3840d79
thomasw21 Document the use of None
e4547612
thomasw21 Woops
150b5a1b
thomasw21 Handle empty document better
3f9efbbd
thomasw21 We might not be able to concat empty tensors
9802ca6f
thomasw21 Handle empty tensor seperately
fcb34b39
thomasw21 Debug
8e43c960
thomasw21 Test
3d004ef9
thomasw21 Add loss masking as script argument
675ef50e
huu4ontocord
huu4ontocord commented on 2021-08-05
thomasw21 thomasw21 assigned thomasw21 thomasw21 4 years ago
thomasw21 thomasw21 added arch&scale
thomasw21 thomasw21 added enhancement
thomasw21 Turns out deepspeed integration of attention matrices prevented dynam…
e304b218
thomasw21 Add more asserts
30fddd22
thomasw21
thomasw21 commented on 2021-08-06
thomasw21
thomasw21 commented on 2021-08-06
thomasw21 Prefix can only see the prefix, it cannot see target
23397ad0
thomasw21 thomasw21 changed the title WIP: Prefix lm Prefix lm 4 years ago
thomasw21 thomasw21 marked this pull request as ready for review 4 years ago
thomasw21 Remove prefix-lm argument as we split the pretrain script
28a712d4
thomasw21
sbmaruf
thomasw21
huu4ontocord
ibeltagy
ibeltagy requested changes on 2021-09-07
ibeltagy
ibeltagy requested changes on 2021-09-15
thomasw21 Merge remote-tracking branch 'origin/master' into prefix_lm
d49d6e5e
thomasw21 Iz PR review
bbfac966
thomasw21 Make masking row dependent when using prefix
0cdb0a94
thomasw21 Revert "Merge remote-tracking branch 'origin/master' into prefix_lm"
a7c51aa1
thomasw21 Make asserts concerning the choice on loss_on_targets_only
6db9b970
thomasw21 Tests (#1)
295e8d0d
thomasw21
ibeltagy
ibeltagy ibeltagy requested a review from ibeltagy ibeltagy 4 years ago
ibeltagy
ibeltagy approved these changes on 2021-09-16
thomasw21
thomasw21 Update code for prefix lm
6a96fb9c
thomasw21 thomasw21 merged 68b46f20 into main 4 years ago
stas00
stas00
stas00
thomasw21
stas00
stas00
thomasw21
stas00
thomasw21
stas00
stas00

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
Labels
enhancement arch&scale
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum