Understanding SOAP from the Perspective of Gradient Whitening
arXiv:2509.22938v1 Announce Type: new Abstract: Shampoo with Adam in the Preconditioner’s eigenbasis (SOAP) has recently emerged as a promising optimization algorithm for neural network training, achieving superior training efficiency over both Adam and Shampoo in language modeling tasks. In this…
