Arista Networks' new 7800R4 modular switches, built on Broadcom's Jericho 3+ ASIC, deliver 576 ports at 800 Gb/sec in a fully loaded 16-line-card chassis and cut power consumption by 65 percent compared to the Jericho 2-based 7800R3. The line card is available now; a variant supporting HyperPort quad-bandwidth aggregation ships in the first quarter of next year.

The move matters because leaf/spine topologies are running out of headroom. AI clusters already commonly span 50,000 to 100,000 accelerators and are pushing toward 1 million or more. A non-blocking two-tier leaf/spine network with 256 leafs and 40 spines tops out at 4,608 ports at 800 Gb/sec. A three-tier leaf/spine/super-spine design exhausts its ports at 27,648 GPUs — and getting there requires 270 super-spines, 540 spines, and 1,728 leafs.

Beyond that ceiling, network builders using currently shipping ASICs from Broadcom, Nvidia, or Cisco must either build modular switches from those chips or keep stacking aggregation tiers, which radically increases hop count and therefore latency between accelerators.

Why go modular again?

The logic echoes the "Magnum" InfiniBand switch that Andy Bechtolsheim, co-founder and chief architect at Arista Networks, created for the Constellation supercomputer eventually acquired by Sun Microsystems and shipped in June 2007. That 3,456-port beast aimed to cut expensive cabling and pack an effective one-hop leaf/spine network inside a single box with huge radix. The analogy to today's GPU machines holds; only the scale has changed.

Brendan Gibbs, vice president of AI, routing, and switching platforms at Arista Networks, frames the tradeoff as buffering versus radix:

"With those two value attributes — the buffering and the port capacity in radix — it is up to customer choice to get the shallow buffer, what I call hierarchical switch architectures — or the deep HBM buffer on package," Gibbs said. "You are only going to need that HBM if you overrun the buffer. So it's like an insurance policy."

He added: "Even if customers don't use the deep buffering of these modular boxes, the port capacity gives them lowest cost infrastructure, the most efficiency."

The economics are straightforward: a modular chassis full of switch chips interlinked on system boards costs less than three or four tiers of leaf-spine-superspine-hyperspine stitched together with electrical and optical cables. PCB traces also fail at a far lower rate than optical transceivers or fiber cables.

What does HyperPort change?

HyperPort — revealed for the Jericho 4 ASIC this summer and now brought back to Jericho 3+ — lashes four 800 Gb/sec ports together to act as a single 3.2 Tb/sec port. Broadcom's numbers:

• 44 percent lower AI job completion time versus ECMP adaptive routing over four independent 800 Gb/sec ports
• One-fourth the number of managed ports for a given bandwidth level
• Up to 70 percent higher link utilization

The Jericho 3+ line cards gain HyperPort support in Q1 next year, well ahead of Jericho 4-based products, which are not expected until roughly this time next year.

What does the R4 family look like?

• 7020R4: fixed-port leaf switches with 10 Gb/sec or 25 Gb/sec downlinks and 100 Gb/sec uplinks; burns 50 percent less power per port than the 7020R3. Both variants ship in Q1.
• 7280R4: 32 ports at 800 Gb/sec, based on Jericho 3+; 20-25 percent lower power than the 7280R3. Available now.
• 7700R4: 18 ports at 800 Gb/sec using the Jericho 3-AI ASIC, scaling to 27,648 GPUs in a leaf/spine network. Already launched.
• 7800R4: modular chassis in four, eight, twelve, or sixteen line-card configurations, linked non-blocking.

The flagship 7800R4 with sixteen line cards drives 1,152 ports at 400 Gb/sec or 576 ports at 800 Gb/sec. Each Jericho 3+ line card exposes 36 ports at 800 Gb/sec with OSFP and QSFP-DD connectors. Arista positions the box for large two-tier Clos networks — the preferred hyperscaler topology — and datacenter interconnect workloads that need massive aggregated bandwidth.

Using a modular chassis as a leaf switch may sound odd, but it is currently the only path to 100,000 XPUs or GPUs and beyond.

What silicon is underneath?

The Jericho 3 ASIC is etched at 5 nanometers at Taiwan Semiconductor Manufacturing Co. It delivers 28.8 Tb/sec of aggregate bandwidth with 304 SerDes running at 100 Gb/sec with PAM-4 encoding: 144 SerDes for downlinks, 160 for uplinks to 51.2 Tb/sec Ramon 3 spine fabric switches. The Ramon 3 also serves as a super-spine layer, letting Arista interlink 32,768 GPUs in a three-tier network.

The Jericho 3-AI, shipping in R4-generation switches for more than a year, adds packet-spraying optimizations tuned for AI workloads. No spec sheet exists for the Jericho 3+, and Arista has not detailed what distinguishes it. All Jericho 3-class chips carry SRAM buffers on-chip plus deep buffers in HBM stacked DRAM on the ASIC package.