transformers
Reset loss to zero on logging in Trainer to avoid bfloat16 issues
#8561

Merged

Reset loss to zero on logging in Trainer to avoid bfloat16 issues #8561

LysandreJik merged 5 commits into huggingface:master from bminixhofer:tr_loss_fix

make tr_loss regular float

c9d7ccfa

Revert "make tr_loss regular float"

d3e8b825

reset loss at each logging step

142d34db

keep track of total loss with _total_loss_scalar

1f40edb0

add remaining tr_loss at the end

ce16f5c7

sgugger approved these changes on 2020-11-17

sgugger requested a review from

LysandreJik 5 years ago

bminixhofer changed the title ~~make tr_loss in Trainer regular float to avoid overflow~~ Reset loss to zero on logging in Trainer to avoid bfloat16 issues 5 years ago

LysandreJik approved these changes on 2020-11-18

LysandreJik merged f6fe41c9 into master 5 years ago

bminixhofer deleted the tr_loss_fix branch 5 years ago

Reviewers

LysandreJik

sgugger

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Reset loss to zero on logging in Trainer to avoid bfloat16 issues #8561 Merged

Reset loss to zero on logging in Trainer to avoid bfloat16 issues #8561

transformers
Reset loss to zero on logging in Trainer to avoid bfloat16 issues
#8561

Merged