Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
arXiv:2502.04416v3 Announce Type: replace Abstract: Scaling large language models (LLMs) improves performance but significantly increases inference costs, with feed-forward networks (FFNs) consuming the majority of computational resources. While Mixture-of-Experts (MoE) architectures can reduce this cost through sparse activation, restructuring existing…
