DeepSpeed
Z3: optimizations for grad norm calculation and gradient clipping
#5504
Merged
Overview
Commits
25
Changes
View On GitHub
Commits
  • z3 scaled_global_grad_norm: repalce get_global_norm with torch.norm
    Nadav Elyahu committed 2 years ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 2 years ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 2 years ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 1 year ago
  • fix grad norm calc in cpu offload and use torch.clip for grad clipping
    Nadav Elyahu committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    lekurile committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    nelyahu committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    nelyahu committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • adding gradient clipping to TestZeroPartialOffloadConfigSweep
    Nadav Elyahu committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    tjruwase committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    GitHub committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
  • Merge branch 'master' into zero3_scaled_global
    loadams committed 1 year ago
FAQ Terms Privacy Refunds Impressum
Loading