llm-foundry
Move loss generating token counting to the dataloader
#1632

Merged

Move loss generating token counting to the dataloader #1632

dakinggg merged 19 commits into mosaicml:main from dakinggg:token-tput

fix

99177b59

try

14751e26

fix

81f61744

9d557980

put back

0f3cbc08

fix

12e4f02a

tensors

4a425b5f

fix

3d207a0a

fix

fd86108a

update

a21c781d

add to ft

b4b9349b

2f75f19b

d92d5352

fix embedding code

120853fd

fix test and clean up;

ad55425a

dakinggg changed the title ~~WIP~~ Move loss generating token counting to the dataloader 1 year ago

a4024228

dakinggg marked this pull request as ready for review 1 year ago

dakinggg requested a review 1 year ago

dakinggg requested a review from

mvpatel2000 1 year ago

dakinggg requested a review from

irenedea 1 year ago

mvpatel2000 commented on 2024-11-03

mvpatel2000 approved these changes on 2024-11-03

pr comments

b0ddad77

7a7055ba

ec2109da

dakinggg merged ce302be4 into main 1 year ago

Reviewers

mvpatel2000

irenedea

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

llm-foundry Move loss generating token counting to the dataloader #1632 Merged

Move loss generating token counting to the dataloader #1632

llm-foundry
Move loss generating token counting to the dataloader
#1632

Merged