On October 5, Huawei's CANN community published a technical document: "Trillion-Parameter MoE Model Pretraining System Reference Practice on the Ascend 950 SuperPoD." According to multiple Chinese tech outlets including 93913 and Observer.com, the DeepSeek pretraining section explicitly covers sharding strategies for 550B and 1.6T models, plus deployment tradeoffs when extrapolating toward the 10T (ten-trillion-parameter) scale — reportedly the first time DeepSeek-related public training material has touched that magnitude.

To put the scale in perspective: DeepSeek-R1 has 671B parameters, so 10T is roughly 15x larger. The current DeepSeek-V4-Pro has 1.6T total parameters (49B active) and V4-Flash has 284B total (13B active), while Alibaba's flagship Qwen 3.8-Max sits at about 2.4T. In other words, 10T is more than four times the largest publicly released Chinese model to date.

What the document covers

Built on a native PyTorch training framework and the Ascend 950 SuperPoD, validated with DeepSeek-V4-Flash training, the reference systematically answers "how do you actually train a trillion-parameter MoE":

  • Multi-level parallelism: CANN provides FSDP (fully sharded data parallel), EP (expert parallel) and CP (context parallel) deployment strategies inside and across supernodes, with optimizations for the memory, communication and compute-efficiency bottlenecks that appear as model size grows;
  • Memory optimization across the board: parameters, gradients, optimizer states and activations each get dedicated memory treatment;
  • New components: the FlexMuon distributed optimizer and distributed training for large-capacity memory tables, aimed at efficiency and stability at extreme scale;
  • Beyond scale: the practice also covers ultra-long sequences, multimodal training and high-availability training scenarios.

The hardware substrate is the Ascend 950 SuperPoD: a 64-card single rack as the basic unit, scaling up to 8,192 cards in high-speed interconnection. At that cluster size, parallel strategy is where the training efficiency lives or dies.

Two sides of the same trillion-parameter coin

At September's Apsara Conference, Alibaba announced its next-gen Qwen would scale to 5-10 trillion parameters (covered here earlier). The interesting contrast: those headlines were about how big to train; this CANN document is about how to train it — sharding, parallel deployment, communication and memory tradeoffs, all engineering detail.

Per NetEase reporting, DeepSeek founder Liang Wenfeng had earlier expressed a philosophy of prioritizing smaller active-parameter models while preserving the ability to extrapolate toward larger scale. The two routes are not in conflict: in MoE architectures, total parameters set the capability ceiling while active parameters set inference cost — a 10T reserve and a cost-efficient small model can advance in parallel.

So what

A 10T model will not ship tomorrow; this reads more like an exploratory engineering reserve. But the signal is clear: as models grow from hundreds of billions toward tens of trillions of parameters, what decides success is no longer peak single-chip compute — it is the training system's parallel strategy, communication efficiency and memory management. That unglamorous systems engineering is precisely the hardest moat to copy. Whether the domestic compute stack can carry the next generation of model scale is a question answered in documents like this one.

Reference: 93913: DeepSeek heading for 10 trillion parameters? Huawei CANN community discloses ultra-large-scale MoE training practice