BigScience Eval Harness #291
Muennighoff
wants to merge
75 commits into
main
from
bseval_harness
Add functionality for running the evaluation harness on single gpu
71728c2f
Add support for pipelining
97a2339c
support tensor parallel
e72a18f5
save the results
0a829651
Minor cleanup
ceddfc5b
Experimental Deepspeed support
c1e80225
Proper deepspeed integration, now working on combined tp and pp
0f8c8c09
Update model loading and clean up code.
37e6962e
Add some options
ee0a1a90
Fix pipelining + fp32 evaluaiton.
cacc58f8
Remove dummy paths in examples/run_evalharness.sh
778f2514
Simplify offline loading with export HF_DATASETS_OFFLINE=1
3d90b18d
Remove accidental copy-paste.
2bb61ac7
Experimantel deepspeed evaluation-path
a362da38
make it work with deepspeed; add instructions
9899be0b
improve
7ef5ba75
make adaptive_seq_len work with deepspeed
9527ad36
move to slurm
d4dacbe9
fixes
151e91ab
cleanup
92123d0a
add instructions on how to import data into the spreadsheet
a6fab1f8
not tracking ppl/em
dedf111e
add task version
a12af5c0
make compatible with lm-eval@master
dbecf81c
switch to 16gb slurm; simplify; improve instructions
c04e3d06
Deepspeed model loading hack
e6e4800f
Restore correct zero state.
5e611bf1
fix conversion script
7937eab0
simpler config
afd38142
Merge remote-tracking branch 'origin/main' into eval_harness
b1a54f33
corrections
9c60079f
add logiqa
d8611374
dealing with custom tokenizers
71587901
fix
f0da71de
Update examples/run_evalharness_deepspeed.md
1e06f41e
Merge branch 'main' into eval_harness
a9221acc
Merge remote-tracking branch 'origin/main' into eval_harness
a7222590
check that the checkpoint path is valid
9ac9fadb
skip --abort_on_unmet_fused_kernel_constraints during eval
8ef90184
disable sanity check on layers-2%pp==0
a798d69f
sort skip_keys
5884dcf3
make the default path unique to avoid overwrite
45bd9c65
Add bootstrap_iters arg
f75e2326
Explain bootstrap_iters flag
7bf75b94
Intermediate results flag
3f18e7be
Add backup file
213317f1
Add arg to reduce bubble for pipeline parallel
1c11b107
Fix adaptive_seq_len via resetting activation shape
f3307058
Extract args.load prior to load_ds_checkpoint_and_setup_megatron
50820352
Parse args prior to loading function to get load_path
db203cce
Add run_evalharness-tr11-176b-ml slurm script
1d6c630a
Add bseval_harness fork compatibility
72447451
Remove superfluous script
6fd4646b
Merge branch 'main' into bseval_harness
e81615e5
Remove duplicates
0214bb77
Remove superfluous string
2ce9ff67
Add emission & example file
1fa06184
Add downloading
9af3e02e
Offload to CPU earlier & increase number of bs in pipleine parallelism
f75af1f9
Add offload arg
9cf7ffd9
add offload arg to slurm scripts
40cf8690
Fix setup_example_logger
93134669
Add torch barrier
d0b2efaa
Add torch barrier
01dc62ac
Improvement
c193ffc3
Be very careful of random states
79cb5693
Woops
bd31b62c
This is already done correctly
c6f76025
Filter out generative tasks
6105fe4a
There's no BOS for bloom
43936d9f
Remove codecarbon
280f1dcd
Merge branch 'main' into bseval_harness
b4660096
Add small model scripts
9a2277ce
Merge branch 'bseval_harness' of https://github.com/bigscience-worksh…
02961ea3
merge main (#331)
472045e6
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub