transformers
3e142cb0 - fix overflow when training mDeberta in fp16 (#24116)

Commit
2 years ago
fix overflow when training mDeberta in fp16 (#24116) * Porting changes from https://github.com/microsoft/DeBERTa/ that hopefully allows for fp16 training of mdeberta * Updates to deberta modeling from microsoft repo * Performing some cleanup * Undoing changes that weren't necessary * Undoing float calls * Minimally change the p2c block * Fix error * Minimally changing the c2p block * Switch to torch sqrt * Remove math * Adding back the to calls to scale * Undoing attention_scores change * Removing commented out code * Updating modeling_sew_d.py to satisfy utils/check_copies.py * Missed changed * Further reduce changes needed to get fp16 working * Reverting changes to modeling_sew_d.py * Make same change in TF
Author
Parents
Loading