MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning
arXiv:2407.20999v4 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks. Typically, LLMs are first pre-trained on large corpora and subsequently fine-tuned on task-specific datasets. However, during fine-tuning, LLMs may forget some…
