Eval harness #212
TevenLeScao
merged
51 commits into
main
from
eval_harness
Add functionality for running the evaluation harness on single gpu
71728c2f
Add support for pipelining
97a2339c
support tensor parallel
e72a18f5
save the results
0a829651
Minor cleanup
ceddfc5b
Experimental Deepspeed support
c1e80225
Proper deepspeed integration, now working on combined tp and pp
0f8c8c09
Update model loading and clean up code.
37e6962e
Add some options
ee0a1a90
Fix pipelining + fp32 evaluaiton.
cacc58f8
thomasw21
commented
on 2021-12-09
Remove dummy paths in examples/run_evalharness.sh
778f2514
Simplify offline loading with export HF_DATASETS_OFFLINE=1
3d90b18d
SaulLu
commented
on 2021-12-14
Remove accidental copy-paste.
2bb61ac7
thomasw21
commented
on 2021-12-22
Experimantel deepspeed evaluation-path
a362da38
stas00
commented
on 2022-01-08
make it work with deepspeed; add instructions
9899be0b
improve
7ef5ba75
make adaptive_seq_len work with deepspeed
9527ad36
move to slurm
d4dacbe9
fixes
151e91ab
cleanup
92123d0a
thomasw21
commented
on 2022-01-09
add instructions on how to import data into the spreadsheet
a6fab1f8
not tracking ppl/em
dedf111e
add task version
a12af5c0
make compatible with lm-eval@master
dbecf81c
switch to 16gb slurm; simplify; improve instructions
c04e3d06
Deepspeed model loading hack
e6e4800f
Restore correct zero state.
5e611bf1
stas00
commented
on 2022-01-14
fix conversion script
7937eab0
simpler config
afd38142
Merge remote-tracking branch 'origin/main' into eval_harness
b1a54f33
corrections
9c60079f
add logiqa
d8611374
dealing with custom tokenizers
71587901
fix
f0da71de
conglongli
commented
on 2022-02-18
stas00
commented
on 2022-02-18
Update examples/run_evalharness_deepspeed.md
1e06f41e
Merge branch 'main' into eval_harness
a9221acc
Merge remote-tracking branch 'origin/main' into eval_harness
a7222590
check that the checkpoint path is valid
9ac9fadb
skip --abort_on_unmet_fused_kernel_constraints during eval
8ef90184
disable sanity check on layers-2%pp==0
a798d69f
sort skip_keys
5884dcf3
TevenLeScao
approved these changes
on 2022-04-26
make the default path unique to avoid overwrite
45bd9c65
Add bootstrap_iters arg
f75e2326
Explain bootstrap_iters flag
7bf75b94
Intermediate results flag
3f18e7be
stas00
commented
on 2022-05-14
Add backup file
213317f1
Add arg to reduce bubble for pipeline parallel
1c11b107
Fix adaptive_seq_len via resetting activation shape
f3307058
Extract args.load prior to load_ds_checkpoint_and_setup_megatron
50820352
Parse args prior to loading function to get load_path
db203cce
Add run_evalharness-tr11-176b-ml slurm script
1d6c630a
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub