Zero Redundancy Training

Slides
Video Lecture

References

  1. ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He2019