Remote JSONL IFT data #275
vchiley
merged
36 commits into
main
from
remote-jsonl-ift
support remote jsonl files for IFT datasets
3f318634
improve docstring
be630c03
add support for other extensions
eded4dfa
dakinggg
commented
on 2023-06-02
don't duplicate validation check
e82ed189
dakinggg
commented
on 2023-06-02
alextrott16
commented
on 2023-06-02
build dataset before tmpdir deletes
4a93eaaa
parse uri
92a14803
dakinggg
commented
on 2023-06-04
only rank 0 download
f6380ecf
only download rank 0
8173998a
dakinggg
approved these changes
on 2023-06-06
alextrott16
approved these changes
on 2023-06-06
better error
0f31e335
break earlier
c3676ff0
log more
67fc6151
more reasonable destination str
171aadb6
use data files format
4d420677
name points to a preprocessing function I guess
455c88a7
debugging
85dcf9bb
always something with HF
edabf535
json vs jsonl [no-ci]
93c50d1c
if hf wants it local, make it local [no-ci]
8b95fae7
back to tempfile [no-ci]
11204e1e
debug
3b9b85aa
debug hfds [no-ci]
b5158ebe
... [no-ci]
a6be0623
don't rename file
ee9f402a
use tempfile again
246f6ff7
Merge branch 'main' into remote-jsonl-ift
1211eae3
Merge branch 'main' into remote-jsonl-ift
bcf5ed62
updt
854e56bb
merge main and cleanup
f39209d1
updt
1f2f9fee
updt
89ee08e7
updt
d7a2f95b
updt
f8a44d97
updt
5aeac479
updt
14f52b51
updt
0139a458
lint
b777d383
alextrott16
approved these changes
on 2023-06-22
Login to write a write a comment.
Login via GitHub