Ampere’s 192-core chips break ARM64 Linux kernel in two-socket systems, company requests higher core count support

Ampere’s new AmpereOne data center CPUs come bursting at the seams with up to 192 cores, but that plentiful helping has caused difficulties with Linux. According to Phoronix, the new CPUs have so many cores that Linux doesn’t support systems when two of Ampere’s 192-core chips (384 total cores) are installed in a single server. For now, the ARM64 Linux kernel only supports systems with 256 cores or less. To fix the issue, Ampere has submitted a patch proposing that the Linux kernel core limit be raised to 512 with a method called “CPUMASK_OFFSTACK.”

This method lets Linux override the default 256-core limit in the current Linux kernel by allocating free bitmaps for CPU masks from memory. This means the Linux core limit can be raised without increasing the kernel image’s memory footprint, as each core adds 8KB to the kernel image size.

Ampere’s new CPUs feature the highest number of cores we’ve seen in a CPU to date. Even AMD’s latest Zen 4c EPYC CPUs don’t quite get there, with the highest core count chip at exactly 128 cores — two chips would hit the limit, but not extend beyond it. That explains why Ampere is the first CPU manufacturer to have serious problems with ARM64 Linux’s 256-core limitation. Thankfully, it won’t affect systems sporting only a single 192-core AmpereOne chip, but it’s a serious problem for data center servers featuring two of these chips in a dual-socket configuration. (SMT logical cores, aka threads, are also well past the 256 figure on various systems, however.)

AmpereOne is a new CPU lineup from Ampere, sporting blisteringly high core counts with models in 136, 144, 160, 176, and 192-core flavors. These chips are built on the ARMv8.6+ instruction set and TSMC’s 5nm node, featuring dual 128b Vector Units, 2MB of L2 cache per core, 3 GHz clock speed, eight-channel DDR5 memory controller, 128 PCIe Gen 5 lanes, and a 200-350W TDP. These chips are designed specifically for high-performance data center workloads that can utilize the hefty core counts.

According to Phoronix, it could take a while before the core count limit gets bumped up to 512. Back in 2021, a patch was sent out proposing the ARM64 CPU core limit be increased to 512, but Linux maintainers denied it because no CPU hardware was available at the time with more than 256 cores. At best, 512-core support won’t be available until 2024, when Linux kernel 6.8 comes out.

This timeline only takes into consideration adding 512-core support the normal way, however, without utilizing the CPU mask off-stack method. Technically, the outgoing Linux kernel already supports the CPU mask off-stack method of increasing CPU core count limits, so it is simply up to Linux maintainers to enable this feature by default.