Dust: Zeroth-Order Pretraining for Transformers Without Backpropagation

Dust introduces a zeroth-order optimization method for pretraining transformers without backpropagation, achieving efficiency and performance comparable to traditional backpropagation. It uses activation-space perturbation and parallel token evaluation, showing scalability and potential to outperform existing methods in compute-rich regimes.

Comments

Read original article