Your MoE Model Does Not Have to Select Fixed Number of Experts

Standard Mixture-of-Experts (MoE) models adopt fixed top-k routing, applying uniform computation across tokens regardless of their complexity. This rigidity often leads to suboptimal efficiency and performance, and dynamic routing could address this by adaptively selecting the optimal number of experts for each token. This post introduces the principles of dynamic routing and reviews key techniques for flexible expert allocation.

What is Dynamic Routing and Why is it Important?

Supposing we have 8 cargo vans and we are running a moving company. The company has a famous regulation: each time we receive an order, we must assign 2 vans to deliver customers' belongings.^[1] Ideally, we should assign vans according to the order size. For example, we can assign 1 van to small orders and more vans to large orders. However, due to the regulation, we have to assign 2 vans to each order, even if the order is small or large.

In the first case (left) of the above figure, it is a waste of money to use 2 vans for a suitcase and a teddy bear. In the last case (right), it may consume more time to deliver the order because the vans may need to make more trips to deliver all the belongings.^[2]

Normally, customers would like to have their belongings delivered cheaply and quickly. So here comes the questions:

1. Could we dynamically assign vans according to the order size?
2. If we could dynamically assign vans, is it possible to reduce the overall cost and time consumption?

These are exactly the problems that dynamic routing aims to address in Mixture-of-Experts (MoE) models. For each token, a conventional router selects top-k experts for computation, where $kk$ is a fixed number. Let's say there are $N_{e}N\_e$ experts in total, the router always selects top-k experts for computation. The token's original representation is denoted as $\mathbf{x}\in {\mathbb{R}}^d\backslash mathbf\{x\}\; \backslash in\; \backslash mathbb\{R\}^d$. Then the token's representation after the MoE layer is computed as:

$$y=\sum _{i=1}^k{G}_i(\mathbf{x})\cdot E_i(\mathbf{x}),y\; =\; \backslash sum\_\{i=1\}^\{k\}\; G\_i(\backslash mathbf\{x\})\; \backslash cdot\; E\_i(\backslash mathbf\{x\}),$$

$$G(\mathbf{x})=\text{Top-}k(\text{softmax}(\mathbf{x}{\mathbf{W}}_g)),G(\backslash mathbf\{x\})\; =\; \backslash text\{Top-\}k(\backslash text\{softmax\}(\backslash mathbf\{x\}\; \backslash mathbf\{W\}\_g)),$$

where $G_i(\mathbf{x})G\_i(\backslash mathbf\{x\})$ is the routing probability of the selected $ii$-th expert, $E_i(\mathbf{x})E\_i(\backslash mathbf\{x\})$ is the selected $ii$-th expert network, and ${\mathbf{W}}_g\in {\mathbb{R}}^{d\times N_e}\backslash mathbf\{W\}\_g\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; N\_e\}$ is the weight matrix for the router.

Briefly, the router network computes the routing probabilities for all experts, and then the token's representation is computed as the weighted sum of the selected experts:

However, different tokens may have different expert selection preferences. For example, for complex reasoning tasks, a token may be forwarded to more experts for better performance. While in simple tasks, a token may be forwarded to fewer experts for better efficiency.

Here, let's transform the above moving company's problems into the MoE context:

1. Could a router dynamically select the number of experts for each token?

• Yes, there are some techniques to dynamically select the number of experts for each token!

2. Does dynamic routing help improve performance and reduce time consumption?

• Yes, dynamic routing could be both effective and efficient!

In this post, we will provide a brief introduction to MoE dynamic routing techniques and their applications in LLMs. Now let's dive into the details!

How to Apply Dynamic Routing?

There are three main categories of dynamic routing techniques:

1. Thresholding: Instead of selecting top-k experts for each token, the router selects experts based on a threshold.
2. Dynamic Proposer: An additional proposer module for predicting the number of activated experts for each token.
3. Zero-Computation Experts: Some special experts are regarded as placeholders for routing, but not used for computation.

Here, we will introduce these three categories of dynamic routing techniques in detail.

Thresholding

Thresholding is the simplest and most straightforward way to apply dynamic routing. It works by setting a threshold for the probability of the experts. If the probability is greater than the threshold, the expert is activated. Otherwise, the expert is deactivated.

Cumulative Thresholding: Huang et al. (2024) [Huang et al., 2024] proposes MoE-Dynamic, which is a thresholding-based dynamic routing technique to select the top-p experts for each token. As the figure below shows, MoE-Dynamic sorts the routing probabilities of experts in descending order, and then selects experts by setting a threshold for the cumulative probability of experts. When the cumulative probability is greater than the threshold, the experts are activated.

However, the routing distribution may tend to be uniform if the model is trained from scratch without any regularization because the model tends to activate more experts to obtain a better performance. To address this issue, the authors propose a regularization loss (probability entropy loss) on the routing probabilities $PP$ to encourage a sharper distribution:

$${\mathcal{L}}_{\text{reg}}=-\sum _{i=1}^{N_e}{P}_i\log P_i\backslash mathcal\{L\}\_\{\backslash text\{reg\}\}\; =\; -\backslash sum\_\{i=1\}^\{N\_e\}\; P\_i\; \backslash log\; P\_i$$

From the experimental results, MoE-Dynamic shows its effectiveness to:

1. achieve better performance than vanilla MoE (~1.68% improvements)
2. accelerate the model training and inference throughput by 5% compared to vanilla MoE
3. show different number of activated experts for different tasks

Trainable Thresholding: Different from cumulative thresholding, DynMoE [Guo et al., 2025] proposes a top-any gating function with gating parameters ${\mathbf{W}}_g\in {\mathbb{R}}^{d\times N_e}\backslash mathbf\{W\}\_g\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; N\_e\}$ and trainable thresholds $\theta \in {\mathbb{R}}^{N_e}\backslash theta\; \backslash in\; \backslash mathbb\{R\}^\{N\_e\}$. To determine if an expert should be activated, DynMoE first computes the cosine similarity $ss$ between the token's representation $\mathbf{x}\backslash mathbf\{x\}$ and the router ${\mathbf{W}}_g\backslash mathbf\{W\}\_g$, then applies a sigmoid function $\sigma (\cdot )\backslash sigma(\backslash cdot)$ to normalize the similarity to [0, 1]. After that, it compares the normalized similarity $\sigma (s)\backslash sigma(s)$ with the trainable thresholds $\sigma (\mathbf{\theta })\backslash sigma(\backslash mathbf\{\backslash theta\})$ to determine if the expert should be activated.

$$s(\mathbf{x})=\frac{⟨\mathbf{x},{\mathbf{W}}_g⟩}{\mathrm{\parallel }\mathbf{x}\mathrm{\parallel }\mathrm{\parallel }{\mathbf{W}}_g\mathrm{\parallel }},s(\backslash mathbf\{x\})\; =\; \backslash frac\{\backslash langle\; \backslash mathbf\{x\},\; \backslash mathbf\{W\}\_g\; \backslash rangle\}\{\backslash |\backslash mathbf\{x\}\backslash |\; \backslash |\backslash mathbf\{W\}\_g\backslash |\},$$

$$G(\mathbf{x})=\text{sign}(\sigma (s(\mathbf{x}))-\sigma \left(\mathbf{\theta }\right)),G(\backslash mathbf\{x\})\; =\; \backslash text\{sign\}\backslash left(\backslash sigma\backslash left(s(\backslash mathbf\{x\})\backslash right)\; -\; \backslash sigma\backslash left(\backslash mathbf\{\backslash theta\}\backslash right)\backslash right),$$

where the number of activated experts is ${\sum }_{i=1}^{N_e}{G}_i(\mathbf{x})\backslash sum\_\{i=1\}^\{N\_e\}\; G\_i(\backslash mathbf\{x\})$.

As the figure below shows, DynMoE can dynamically select the number of activated experts by setting different thresholds for different experts.

To enable diverse routing and suppress uniform routing, DynMoE also introduces a regularization loss term, including a diverse loss to lower the similarities within routers and a simplicity loss to maintain numerical stability.

$${\mathcal{L}}_{\text{reg}}=\underset{\text{diversity loss}}{\underbrace{{\parallel {\mathbf{W}}_g^T{\mathbf{W}}_g-{\mathbf{I}}_{N_e}\parallel }_2}}+\underset{\text{simplicity loss}}{\underbrace{\frac{1}{N_e}\sum _{i=1}^{N_e}{\parallel {\mathbf{w}}_{g,i}\parallel }_2}},\backslash mathcal\{L\}\_\{\backslash text\{reg\}\}\; =\; \backslash underbrace\{\backslash left\backslash |\backslash mathbf\{W\}\_g^T\; \backslash mathbf\{W\}\_g\; -\; \backslash mathbf\{I\}\_\{N\_e\}\backslash right\backslash |\_2\}\_\{\backslash text\{diversity\; loss\}\}\; +\; \backslash underbrace\{\backslash frac\{1\}\{N\_e\}\; \backslash sum\_\{i=1\}^\{N\_e\}\; \backslash left\backslash |\backslash mathbf\{w\}\_\{g,i\}\backslash right\backslash |\_2\}\_\{\backslash text\{simplicity\; loss\}\},$$

where ${\mathbf{I}}_{N_e}\backslash mathbf\{I\}\_\{N\_e\}$ is the identity matrix of size ${\mathbb{R}}^{N_e\times N_e}\backslash mathbb\{R\}^\{N\_e\; \backslash times\; N\_e\}$.

The results demonstrate that DynMoE can achieve comparable performance than vanilla MoE with 11% throughput improvements.

Non-Linear Thresholding: Wang et al. (2025) [Wang et al., 2025] proposes ReMoE, which is a non-linear thresholding-based dynamic routing technique to select experts for each token. Instead of setting a threshold as a hyper-parameter, ReMoE finds activated experts by replacing the Top-\(k\) softmax with a ReLU activation function.

$$G(\mathbf{x})=\text{ReLU}(\mathbf{x}{\mathbf{W}}_g),G(\backslash mathbf\{x\})\; =\; \backslash text\{ReLU\}(\backslash mathbf\{x\}\; \backslash mathbf\{W\}\_g),$$

Due to the non-linearity of ReLU, experts with smaller activations would be filtered out, and the number of activated experts is: ${\sum }_{i=1}^{N_e}\mathbb{1}\{G_i(\mathbf{x})>0\}\backslash sum\_\{i=1\}^\{N\_e\}\; \backslash mathbb\{1\}\backslash \{G\_i(\backslash mathbf\{x\})\; >\; 0\backslash \}$. As shown in the figure below, due to the ReLU activation, experts with positive routing logits would be activated. Besides, the ReLU gate is fully differentiable, which allows a stable training process.

To constrain the number of expected activated experts, ReMoE introduces a sparsity regularization loss term:

$${\mathcal{L}}_{\text{reg}}(t)=\lambda (t)\cdot \sum _{i=1}^{N_e}{G}_i(\mathbf{x}),\backslash mathcal\{L\}\_\{\backslash text\{reg\}\}(t)\; =\; \backslash lambda(t)\; \backslash cdot\; \backslash sum\_\{i=1\}^\{N\_e\}\; G\_i(\backslash mathbf\{x\}),$$

$$\lambda (t)=\lambda (t-1)\cdot {\alpha }^{\text{sign}((1-N_k\mathrm{/}{N}_e)-S(t))},\backslash lambda(t)\; =\; \backslash lambda(t-1)\; \backslash cdot\; \backslash alpha^\{\backslash text\{sign\}((1\; -\; N\_k/N\_e)\; -\; S(t))\},$$

$$S(t)=1-\frac{1}{N_l{N}_e}\sum _{l=1}^{N_l}\sum _{i=1}^{N_e}\mathbb{1}\{G_i^l(\mathbf{x})>0\},S(t)\; =\; 1\; -\; \backslash frac\{1\}\{N\_l\; N\_e\}\; \backslash sum\_\{l=1\}^\{N\_l\}\; \backslash sum\_\{i=1\}^\{N\_e\}\; \backslash mathbb\{1\}\backslash \{G\_i^l(\backslash mathbf\{x\})\; >\; 0\backslash \},$$

where $\lambda (t)\backslash lambda(t)$ is an adaptive coefficient at training step $tt$, which is initialized with a small value. $\alpha >1\backslash alpha\; >\; 1$ is a hyper parameter to control the update multiplier, and $S(t)S(t)$ represents the sparsity of the activated experts at training step $tt$. Here the sparsity is computed over $N_{l}N\_l$ layers. The whole regularization loss is designed to close the gap between the number of expected activated experts $N_{k}N\_k$ and the actual number of activated experts. Therefore, ReMoE does not bring training or inference acceleration, but it can adaptively adjust the number of activated experts on each layer to improve the final performance.

BlockFFN [Song et al., 2025] further improves the ReLU activation with an RMS layer normalization to control the magnitude of the activations and stabilize the training process.

$$G(\mathbf{x})=\text{RMSNorm}(\text{ReLU}(\mathbf{x}{\mathbf{W}}_g))\mathrm{.}G(\backslash mathbf\{x\})\; =\; \backslash text\{RMSNorm\}(\backslash text\{ReLU\}(\backslash mathbf\{x\}\; \backslash mathbf\{W\}\_g)).$$

BlockFFN also proposes an activation locality loss to encourage similar activation patters in neighboring tokens, and a chunk sparsification loss to maximize the sparsity of a chunk of consecutive tokens. Based on these settings and specially designed kernels, BlockFFN achieves 3.67x speedup over baseline and shows better commonsense reasoning ability compared to ReMoE especially on larger models.

Dynamic Proposer

Intuitively, dynamic routing aims to change the value of $kk$ for each token. Besides the above thresholding-based ideas, can we directly predict the value of $kk$? The answer is YES, and that is exactly the idea of dynamic proposer. Dynamic proposers take the token's representation as input and output the number of activated experts as a prediction.

Ada-K [Yue et al., 2025] applies an additional linear projection to the token's representation to obtain a probability distribution and then samples the activated expert number $kk$ from the distribution.

$$P(\mathbf{x})=\text{softmax}(\mathbf{x}{\mathbf{W}}_p),P(\backslash mathbf\{x\})\; =\; \backslash text\{softmax\}(\backslash mathbf\{x\}\; \backslash mathbf\{W\}\_p),$$

$$k\sim P(\mathbf{x}),k\; \backslash sim\; P(\backslash mathbf\{x\}),$$

where ${\mathbf{W}}_p\in {\mathbb{R}}^{d\times N_e}\backslash mathbf\{W\}\_p\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; N\_e\}$ is a trainable parameter. The figure below demonstrates the sampling process for training. After clicking on the Sample k button, Ada-K would sample the $kk$ value based on the probability distribution.

Because such a sampling process is not differentiable, Ada-K updates ${\mathbf{W}}_p\backslash mathbf\{W\}\_p$ with the Proximal Policy Optimization (PPO) algorithm [Schulman et al., 2017], and take the log likelihood of language modeling as the reward:

$$r=\log P(x_t\mathrm{\mid }{x}_1,\dots ,x_{t-1}),r\; =\; \backslash log\; P(x\_t\; |\; x\_1,\; \backslash dots,\; x\_\{t-1\}),$$

where $x_{t}x\_t$ is the expected token to be generated.

The sampling process is useful for the training process since it brings stochasticity to explore different efficiency-performance trade-offs and compute policy gradients for PPO. During inference, Ada-K simply selects $kk$ with the highest probability as the activated expert number.

For pre-trained language models, Ada-K is efficient since it only updates the linear projection matrix ${\mathbf{W}}_p\backslash mathbf\{W\}\_p$ and keeps all other parameters frozen. For example, when applying Ada-K on Qwen1.5-MoE-14.3B-A2.7B^[3], it takes 1.58h to fine-tune 2.95M parameters, and Ada-K could bring 1.22x speedup over the baseline.

Zero-Computation Experts

Zero-Computation Experts constitute an efficient sparse-expert mechanism within dynamic routing. The basic idea is to incorporate a subset of experts that perform no actual computations during the forward pass. By maintaining a fixed number of activated experts, the selection of these zero-computation experts introduces neither additional FLOPs nor memory overhead, thereby effectively reducing the computational cost. This design improves efficiency without compromising model capacity, and it further enables flexible, adaptive resource allocation when combined with dynamic routing.

Null Experts: Zeng et al. (2024) [Zeng et al., 2024] propose AdaMoE, in which zero-computation experts are defined as performing no operations and consuming zero FLOPs. Building on the original top-k MoE, the router still selects a fixed number of experts, but some of these may be zero-computation null experts. When a token is routed to a null expert, it is dropped out and no forward computation is performed, no additional FLOPs are incurred, and no activation memory is consumed, allowing the number of real experts per token to adapt dynamically.

Following the introduction of zero-computation experts, the lack of a load-balancing mechanism may lead the router to over-select real experts, resulting in an uneven workload distribution and undermining the intended FLOPs reduction. To mitigate this, AdaMoE incorporates an average-load strategy for null experts within the load-balancing loss:

$${\mathcal{L}}_{\text{balance}}=(N_e+N_z)\sum _{i=1}^{N_e+N_z}{\tilde{f}}_i\cdot P_i,\backslash mathcal\{L\}\_\{\backslash text\{balance\}\}\; =\; (N\_e\; +\; N\_z)\; \backslash sum\_\{i=1\}^\{N\_e\; +\; N\_z\}\; \backslash tilde\{f\}\_i\; \backslash cdot\; P\_i,$$

where $N_{z}N\_z$ denotes the number of null experts, $f_{i}f\_i$ denotes the load of the $ii$-th expert and $P_{i}P\_i$ denotes the routing probability of the $ii$-th expert. The modified load-balancing loss in AdaMoE accounts for null experts to control the average load of real experts, while avoiding unnecessary constraints among identical zero-computation experts, thus giving the router greater flexibility to optimize token-to-expert allocation.

From the experimental results, AdaMoE demonstrates that it achieves higher average accuracy across multiple benchmarks while reducing FLOPs by an average of 15.21% during inference.

Zero, Copy, and Constant Experts: Jin et al. (2025) [Jin et al., 2025] propose MoE++, a heterogeneous MoE framework designed to improve both efficiency and effectiveness. It introduces three types of zero-computation experts: Zero, Copy, and Constant experts, which replace a subset of real experts to reduce computational cost without altering expert counts or model capacity:

• Zero Experts: output a zero vector, requiring no computation: $E_i(\mathbf{x})=\mathbf{0}E\_i(\backslash mathbf\{x\})\; =\; \backslash mathbf\{0\}$
• Copy Experts: reuse the input token representation directly: $E_i(\mathbf{x})=\mathbf{x}E\_i(\backslash mathbf\{x\})\; =\; \backslash mathbf\{x\}$
• Constant Experts: output a learned constant vector independent of input: $E_i(\mathbf{x})={\alpha }_1\mathbf{x}+{\alpha }_2\mathbf{v}E\_i(\backslash mathbf\{x\})\; =\; \backslash alpha\_1\; \backslash mathbf\{x\}\; +\; \backslash alpha\_2\; \backslash mathbf\{v\}$, where $[{\alpha }_1,{\alpha }_2]=\text{softmax}(\mathbf{x}{\mathbf{W}}_c)[\backslash alpha\_1,\; \backslash alpha\_2]\; =\; \backslash text\{softmax\}(\backslash mathbf\{x\}\; \backslash mathbf\{W\}\_c)$. ${\mathbf{W}}_c\in {\mathbb{R}}^{d\times 2}\backslash mathbf\{W\}\_c\; \backslash in\; \backslash mathbb\{R\}^\{d\; \backslash times\; 2\}$ and $\mathbf{v}\in {\mathbb{R}}^d\backslash mathbf\{v\}\; \backslash in\; \backslash mathbb\{R\}^d$ are trainable parameters.

These experts incur negligible computation, communication, and memory overhead. When these zero-computation experts are selected, they effectively skip real expert computation while maintaining the top-k selection budget and sparsity patterns.

To address potential issues such as unstable routing or over-selection of low-computation experts, MoE++ introduces a Heterogeneous Load Balance Loss, controlled by a hyperparameter $\tau \backslash tau$ which regulates the allocation of tokens between zero-computation and FFN experts, promoting a balanced computational load across the expert ensemble.

The experimental results show that MoE++ consistently outperforms same-scale Vanilla MoE across downstream tasks, improves expert throughput by 15%, and achieves performance comparable to dense models with 2–3x more parameters under limited training budgets.

Challenges

Dynamic routing techniques have been utilized to various MoE models. For example, Meituan LongCat Team has scaled Copy Expert (one of the zero-computation experts) MoE model to 560B parameters [Meituan LongCat Team, 2025], and Uni-MoE-2.0-Omni employs thresholding-based dynamic routing to build multi-modal MoE. [Li et al., 2025]

However, there are still some challenges to be addressed for dynamic routing:

• Performance-Efficiency Tradeoff: Although some dynamic routing techniques can achieve better performance together with efficiency, the performance and efficiency are still a tradeoff. For example, DynMoE brings 11% throughput improvements over vanilla MoE, but the performance is slightly lower. This is because the performance and efficiency are not always aligned. If we want to achieve extreme sparsity, the number of activated experts would be very small, which may lead to performance degradation.
• Efficient Implementations: For heterogeneous zero-computation experts or special thresholding-based dynamic routing, current grouped GEMM kernels may not be directly applicable, and specialized implementations are needed. For example, BlockFFN designs kernels for ReLU activation and chunk-level sparsification, which significantly accelerates the model throughput.
• Sparsity Controlling: If no sparsity regularization is applied, the number of activated experts tends to be very high due to the performance optimization objective. Therefore, most dynamic routing techniques have to apply the sparsity regularization to control the number of activated experts. However, it is still challenging to control sparsity with the same level of performance as the vanilla top-k strategy.
• Expert Load Balancing: For MoE with zero-computation experts, models may tend to select regular experts, which may lead to uneven workload distribution and undermining the intended FLOPs reduction. To address this, dynamic routing should incorporate an load balancing strategy for both regular and zero-computation experts.

Conclusion

In this post, we provide a brief introduction to MoE dynamic routing techniques and review several key methods, including thresholding, dynamic proposer, and zero-computation experts. Dynamic routing offers a promising direction for MoE models by breaking the rigidity of fixed top-k routing and enabling adaptive expert allocation for each token. While challenges such as performance-efficiency tradeoff, efficient implementations, sparsity controlling, and expert load balancing still exist, dynamic routing has shown great potential in improving the performance and efficiency of MoE models, and has been applied to existing large language models. As MoE models continue to evolve, we expect dynamic routing to play an increasingly important role in future large-scale foundation models.

Footnotes

[1] Similar to Top-2 expert selection out of 8 ones in MoE models, and each order represents a token.

[2] More trips means more time consumption if no stuff is abandoned due to limited space, similar to disabling token dropping in MoE training. If tokens are dropped, there may induce a performance degradation. [Gale et al., 2023]

[3] https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B

References

[Huang et al., 2024] Huang, Q., An, Z., Zhuang, N., Tao, M., Zhang, C., Jin, Y., Xu, K., Chen, L., Huang, S., & Feng, Y. (2024). Harder Task Needs More Experts: Dynamic Routing in MoE Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12883–12895.

[Guo et al., 2025] Guo, Y., Cheng, Z., Tang, X., Tu, Z., & Lin, T. (2025). Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models. In The Thirteenth International Conference on Learning Representations (ICLR).

[Wang et al., 2025] Wang, Z., Zhu, J., & Chen, J. (2025). ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing. In The Thirteenth International Conference on Learning Representations (ICLR).

[Song et al., 2025] Song, C., Zhao, W., Han, X., Xiao, C., Chen, Y., Li, Y., Liu, Z., & Sun, M. (2025). BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity. In Second Conference on Language Modeling (COLM).

[Yue et al., 2025] Yue, T., Guo, L., Cheng, J., Gao, X., Huang, H., & Liu, J. (2025). Ada-k routing: Boosting the efficiency of moe-based llms. In The Thirteenth International Conference on Learning Representations (ICLR).

[Schulman et al., 2017] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

[Zeng et al., 2024] Zeng, Z., Miao, Y., Gao, H., Zhang, H., & Deng, Z. (2024). AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 6223–6235.

[Jin et al., 2025] Jin, P., Zhu, B., Yuan, L., & Yan, S. (2025). MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts. In The Thirteenth International Conference on Learning Representations (ICLR).

[Meituan LongCat Team, 2025] Meituan LongCat Team. (2025). Longcat-flash technical report. arXiv preprint arXiv:2509.01322.

[Li et al., 2025] Li, Y., Chen, X., Jiang, S., Shi, H., Liu, Z., Zhang, X., Deng, N., Xu, Z., Ma, Y., Zhang, M., et al. (2025). Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data. arXiv preprint arXiv:2511.12609.

[Gale et al., 2023] Gale, T., Narayanan, D., Young, C., & Zaharia, M. (2023). Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5, 288–304.