sd-scripts
Amazing Flux FFT quality on a 5090: (N)AdamW / Adan with fused backwards pass & CPU offloading
#2187
Open
Open
Amazing Flux FFT quality on a 5090: (N)AdamW / Adan with fused backwards pass & CPU offloading #2187
araleza
wants to merge
8 commits into
kohya-ss:sd3
from
araleza:sd3-adamw-offload
Support for fused (N)AdamW + Kahan + momentum offloading FFT on a 5090.
225ea362
Removed mention of Adafactor from warning message as this situation c…
f583e356
Fixed fused_backward_pass error message as it was not accessing the c…
c7b62f74
The optimizer type check in a warning message was incorrect
1cf1f2b9
Changed cpu storage of exp_avg[_sq] from bf16 to powed/scaled u16
5ea1ada2
Changed u16 pow() factor from 16.0 to 8.0, seems to better distribute…
65781334
Now exp_avg[_sq] are stored on cpu in 24 bit format. Also changed som…
f6f3d6e3
Added Adan offloading optimizer, fp32 params, and 'cautious' updates
da17be08
araleza
changed the title Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub