Mixture-of-Experts gave large language models the freedom to scale parameters via sparse activation, but inference has long rested on an unchallenged assumption: the expert is the smallest unit of scheduling. Whichever experts the router picks, each expert network runs in full—no more, no less. PCoMoE (arXiv:2609.01024), accepted to the EMNLP 2026 Main Conference, takes that assumption apart. The authors argue that experts contain substantial reusable computational structure, and what inference should really schedule is not the whole expert, but fine-grained paths inside it.

The Problem: The Whole Expert Is a "Shipping Container"

The paper's diagnosis is blunt. Modern MoE inference frameworks—whether they manage, schedule, or prune experts—treat experts as atomic execution units. This has two consequences. First, the optimization boundary gets fixed too early: optimizing at expert granularity leaves intra-expert computational redundancy permanently invisible. Second, expert computation in MoE naturally contains shareable intermediate structures, but since frameworks only recognize whole experts, those reuse opportunities are all wasted.

A logistics analogy: today's MoE inference bills by the whole container, even when two shipments could share one box.

The Approach: Decompose Experts into Composable Paths

PCoMoE is a path-compositional execution framework with three design layers:

  • Path-level formulation: expert computation is formalized as fine-grained compositional paths. Per the official repository, experts are decomposed into reusable expansion-side and projection-side components;
  • Compatibility-aware layer-wise pruning: suppresses low-value path combinations and keeps combinatorial explosion in check;
  • A hardware-friendly execution engine: exploits reusable sub-expert structures while keeping overhead strictly bounded.

The GitHub repository also highlights a key design element: source-grouped compute reuse, which groups computation by source so that multiple activated experts share underlying work.

The Results: Faster and More Accurate

The numbers reported in the paper: up to 1.31x end-to-end inference speedup, alongside a 10% improvement in model accuracy. Speed and accuracy usually trade off—pruning for speed at the cost of quality is the norm—yet PCoMoE reports improvement in both directions. A mechanistic explanation: once low-value paths are pruned, the model suffers less interference from "bad combinations." That said, these are author-reported numbers; independent replication awaits community verification.

One caveat: the official code repository (github.com/gzyyy0/PCoMoE) is live, but currently contains only paper information and acknowledgments. The implementation is marked as "being organized, will be released soon"—practical adoption will have to wait.

So What

The battle over MoE inference optimization is moving from the framework layer into the structural layer. Prior work mostly focused on routing strategy and expert parallelism; PCoMoE cuts into the expert itself and offers a finer-grained execution abstraction. The value of this direction: sparse activation has proven to be a viable way to scale MoE capacity, and as models keep growing, the redundancy wasted inside each expert gets multiplied accordingly. For inference engine teams, this is a new front; for researchers, the granularity of the execution unit is itself a design variable worth re-examining. Full paper: arXiv:2609.01024.