xAI Colossus: Spectrum-X Ethernet for a 100,000-GPU training cluster

Training one model across 100,000 GPUs needs a network where many large flows run at once without collisions. NVIDIA says standard Ethernet at this scale produces thousands of flow collisions and reaches only 60 percent of possible data throughput.

Data Center Frontier reads the 19-day GPU deployment as a sign of xAI's focus on scaling its infrastructure quickly; xAI's own goals are not stated in the cited sources.¹²

Colossus started with 100,000 NVIDIA Hopper GPUs in Memphis, Tennessee, in a former Electrolux factory according to Data Center Frontier, which also says Dell Technologies and Supermicro partnered with xAI on the build. Its RDMA network uses NVIDIA Spectrum-X Ethernet: Spectrum SN5600 switches (Spectrum-4 ASIC, ports up to 800 Gb/s) paired with BlueField-3 SuperNICs.

Data Center Frontier reports liquid-cooled Supermicro servers with eight GPUs each, eight servers per rack, and a 400 GbE network interface per GPU plus one more per server. At the time of NVIDIA's release xAI was doubling the cluster toward 200,000 Hopper GPUs.

The same article notes cost, power and cooling concerns for a system of this size; the Tennessee Valley Authority approved more than 100 MW of power for the site.¹²

What others can learn: at very large GPU counts the network, and its congestion control, decides how much of the GPU spend turns into training work, so it should be designed with the cluster rather than added later. Limits: very few organizations build at this scale, and few buyers can secure this much hardware at once. Every performance figure comes from NVIDIA; the 60 percent baseline is NVIDIA's own description of standard Ethernet, not a measured comparison on this site, and no test method is published. Clusters of a few hundred GPUs often run well on standard Ethernet or InfiniBand fabrics without these features.