Validation #875
XiaohanZhangCMU
merged
78 commits into
mosaicml:byod/data_validation
from
XiaohanZhangCMU:validation
add validation script
8cb65225
update
c59c11fd
change token count function
66f34ebd
reorganize cells
2cd387b5
Add unit tests
3eac3bf1
Add a printout for CPT
d2d97677
update question
be255917
Add questions
4651be7d
Fix lints
5cd6a94d
Merge branch 'main' into validation
8e2c1f4b
update format
e6e4a81f
Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX …
34c56904
update
1668b9a4
nb source
22191350
add validation script
86c6e87a
update
678b3766
change token count function
297e057d
reorganize cells
09d0ebbd
Add unit tests
460df65c
Add a printout for CPT
3ffd2003
update question
9362886a
Add questions
898e5acf
Fix lints
a4bef711
update format
4ca9cc6c
update
d636a0f8
nb source
827d1551
Remove license insert for validation notebook
6bbf3fce
Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX …
4f6a4fb3
Add validation utils
5966b68b
Merge branch 'main' into validation
da178136
Minor cleanups (#858)
a7c36bcc
update utils/__init__.py to include extra validation functions
55e46265
update notebook
45544a1e
update
d2797b35
Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX …
019da779
update
756fdae2
Read UC delta table (#773)
6de8c37f
Add download remote function to util
93b5a9f9
update
b47c878a
remove fused layernorm (#859)
fa8f3d96
update
13fd34c7
update
610f6694
update
9f2e51b3
update
ec68f10f
update
1e76068e
update
7a5c164f
Merge branch 'main' into validation
e76038fc
update
5b413f55
update
a1aa31f1
update
d24fd5ce
Remove hardcoded combined.jsonl with a flag (#861)
da3bea14
bump (#828)
936e3a1b
Add dask and dataframe_to_mds
55fce374
update
86e2412e
update
bbfec65d
update
b2e880de
update
596443af
Add notebook
ea651873
update
378a4e09
update
af6e9aa9
remove script and tests, keep notebook
4e286ecd
update
09c4892c
update
c82da6cc
update
e5f83cc5
update
17d2b9fd
Merge branch 'main' into validation
6579d552
Merge branch 'byod/data_validation' into validation
56308ffa
Always initialize dist (#864)
6517a307
updated notebook
4daa3248
Merge branch 'main' into validation
b809691f
remove scripts keep notebook
8b75f946
merge with byod/data_validation
99bf2cd5
update notebook. rephrase.
22014d67
merged
d9f28aac
update
43c8ac9d
Add response tokens
b8ac771e
update
1b9681c0
merge
16883c22
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub