A Packaging Timeout That Brought a CPU to the Bench

In February 2026, while cross-compiling normaliz for the loong13 (Debian 13 port to LoongArch), maintainer Wang Miao hit a strange failure: the math library's self-test looped indefinitely on a LA664-based Loongson server. The first instinct was to blame weak memory ordering inside normaliz itself, or a bug in the OpenMP runtime's #pragma omp atomic. After reading the source end-to-end and shrinking the test program down repeatedly, the issue refused to go away.

The fact that simple atomic-add microbenchmarks could not reproduce the issue meant the problem was not in the application layer. Looking at the disassembly, the compiler turned #pragma omp atomic into a LoongArch64 amadd.d—the very instruction supposed to guarantee atomicity for atomic adds.

Six months later, Wang Miao and jia.je changed tack: let AI search for a minimal reproducer while they steered the investigation. Within two days, AI's attention locked onto a memcpy call inside the normaliz processing function, which in turn pointed at the LASX-vectorised memory-copy path that glibc had enabled on LA664.

Three Legs of the Trigger

When the CPU bug is unpacked, the trigger conditions turn out to be very specific:

  • The two threads must run on different physical cores—two SMT siblings on the same core do not trigger it.
  • Both threads perform atomic operations without a data barrier on the same address—e.g. amadd.d rather than amadd_db.d.
  • At least one of the threads interleaves vectorised memory loads between atomic operations.

In the team's minimal reproducer, each normaliz data point is exactly 2208 bytes (276 uint64_ts), which sits inside one LASX 256-bit register window. Two threads each slice the list, run a vectorised memcpy first, then a relaxed atomic-add, and finally compare their counters at the end of the round. The reproduction is startlingly stable: on a 3C6000/S with two physical cores and 30 trials, the "both sides doing LASX reads" configuration triggers near 100% of the time; amcas_db.d (with the data-barrier suffix) under the same conditions fails 0%.

Rust Did Not Escape Either

What is more embarrassing is that the Rust standard library fell into the same trap. Both the reference-counting in std::sync::Arc and the Sender clone inside std::sync::mpsc lower to barrier-less amadd. The authors built a safe Rust program that, using Arc or mpsc, crashed with SIGABRT or glibc heap-corruption reports—the classic signature of a use-after-free.

In practical security terms, however, the exploitability is narrow: both threads must operate on the same in-process counter with relaxed atomicity, and at least one of them must run vectorised memory reads. That trigger condition is naturally fenced off by process isolation, so it is hard to weaponise one-sided.

Loongson's 14-Day Response

Email sent on 26 August, test firmware delivered on 9 September. The fix is surprisingly lightweight: set bit 13 of an undocumented internal CSR, MCSR24. Single-core performance is unchanged; multi-core performance drops only slightly—well within usable range.

There is also a software-side mitigation: compilers could be patched to emit am<OP>_db.* instead, or to fall back to LL/SC loops. The catch is that every existing LA664 binary would need to be recompiled. That path obviously does not fit inside a 14-day window, whereas the CSR tweak takes a single command.

The affected scope is limited to chips using the LA664 core, namely the 3C6000/S and 3A6000 families. The earlier LA464 (3A5000) is unaffected. Loongson promised to publish the firmware before the National Day window (1 October).

What the AI Actually Did

Looking back, the most interesting part of this story is not Loongson's 14-day turnaround, but what role AI played in the six months of debugging.

In the first round, humans had largely concluded that amadd was losing updates, but they could not reduce the entire normaliz codebase to a minimal reproducer. What AI did in the second round was, in effect, narrow the code-search space from "compiler theory / memory ordering / OpenMP runtime" down to "glibc memcpy implementation choice," and then, using properties of the xvld instruction, assemble a stable minimal reproducer. This is exactly the kind of "code archaeology" that working programmers dislike.

But AI did not finish the job on its own. The crucial hint that explicitly told AI the atomic-add was broken, and that pushed it toward memcpy, still came from the developers. In other words, AI's contribution is "rapidly verifying known hypotheses across large codebases and pinning them down to a single line," whereas the decisive jump—"this is a CPU bug, not a software bug"—remains a human engineer's job.

A New Footnote on an Old Topic

CPU errata are everywhere: ARM's public-cores Software Developer Errata Notice has plenty, Intel and AMD each have dozens to hundreds of documented errata, and only a handful of them really affect daily use. LoongArch is a young architecture whose ecosystem is still growing. The fact that an ordinary packaging timeout could be traced all the way to a CSR-level fix says two things: the user community is testing seriously, and the vendor is responding seriously.

Next time you see a baffling multi-threaded crash on LA664, the first thing to check is whether glibc's memcpy has LASX enabled, and whether the compiler is emitting barrier-less atomics. The combination of those two factors probably explains nine out of ten "code looks right but the run is unstable" mysteries on LoongArch.