Each few years, a brand new class of workload arrives that breaks the assumptions of the earlier technology and forces us to rethink not simply the structure, however the underlying physics of how bits transfer. Generative AI is one such second.
What makes this inflection level completely different is that we’re not simply asking networks to hold extra site visitors. We’re asking them to maintain a tightly synchronized, latency-sensitive compute setting that spans a number of knowledge facilities separated by tens or lots of of kilometers. That’s a essentially new downside, and fixing it requires co-innovation throughout silicon, techniques, and optics in methods the {industry} hasn’t tried earlier than.
Extra GPUs, extra intelligence
There’s a easy and profound reality on the coronary heart of the AI period: extra GPUs unlock extra intelligence. Each important leap in AI functionality over the previous six years—from language fashions that would full a sentence to techniques that may purpose, code, and create—has been pushed by coaching on bigger clusters of GPUs. Scaling from lots of of GPUs to lots of of 1000’s has been instrumental in getting us right here.
However that trajectory has run into a tough bodily wall: energy. A single high-performance GPU attracts 700 watts (W) or extra at full load. A rack of GPU servers attracts 80 to 150 kilowatts (kW). A coaching cluster giant sufficient to develop a frontier AI mannequin can eat 10 to fifteen megawatts (MW), roughly equal to a small city’s electrical energy demand. And essentially the most superior fashions being educated right now require clusters that method or exceed 100 MW at a single website, representing 60,000 to 70,000 GPUs or extra.
At this scale, energy has develop into the binding constraint. Energy availability and price in densely populated areas, mixed with the sheer magnitude of electrical energy required, implies that the biggest AI coaching clusters have outgrown what any single facility can help. Knowledge facilities are migrating to less-populated areas with cheaper vitality, making interconnection of GPUs throughout knowledge facilities a prerequisite. When the GPUs wanted to coach the following technology of AI are unfold throughout two or extra websites, the community connecting them should carry out as in the event that they had been in the identical room. This is why scale-across exists.
The bandwidth hierarchy: From DCI to scale-across
To grasp this new problem, it helps to hint how knowledge heart bandwidth necessities have developed. Every technology has been extra demanding by orders of magnitude.
Conventional knowledge heart interconnect (DCI) set the baseline. DCI joins knowledge facilities to different knowledge facilities and finish customers over wide-area networks. It was constructed for redundancy, geographic attain, and enterprise workload distribution.
Entrance-end networks emerged subsequent to deal with site visitors between customers, functions, and cloud providers—video streaming, social media, cloud-native functions—at roughly 7x the bandwidth of DCI.
The actual step change, scale-up networks, emerged with AI. As knowledge facilities pivoted from general-purpose compute to AI powerhouses, normal servers gave option to GPUs and specialised accelerators. Inside a rack, these units are interconnected in scale-up domains at roughly 504x the bandwidth of DCI—related by high-speed copper at 100 to 200 Gbps per lane throughout distances of as much as 3 meters (m), showing to the software program stack as a single logical compute unit.
Scale-out networks then prolonged the AI cloth throughout a complete knowledge heart, connecting racks of GPUs at roughly 56x DCI bandwidth by high-speed Ethernet and InfiniBand switching materials. As soon as distances develop past a number of meters—spanning rack rows and knowledge heart flooring at reaches of 100 meters to 2 kilometers (km)—copper can not keep sign high quality at these speeds, and pluggableome . Consequently, applied sciences like co-packaged optics and linear pluggable optics emerged to handle the ability and density penalties of deploying optics at this scale.
And now we arrive at scale-across, the frontier the place the physics get genuinely laborious.
Scale-across networking: The promise—and the problem
Scale-across is the reply to the geographically distributed GPU downside, and it’s not merely DCI with greater bandwidth. Conventional DCI connects CPUs throughout knowledge facilities and to finish customers, dealing with many low-bandwidth, loss-tolerant, asynchronous flows that develop linearly. Scale-across connects GPUs and scale-out networks, carrying a small variety of extraordinarily high-bandwidth, loss-intolerant, synchronous, long-lived flows that can’t tolerate dropped packets or timing mismatches with out forcing a full restart of the AI job. And people flows are rising exponentially.
The dimensions distinction alone is placing. Scale-across networks require someplace between 12,000 and 32,000 ports—and context makes clear why. A 100 MW knowledge heart homes roughly 60,000 to 70,000 GPUs, every producing as much as 800 Gbps of back-end site visitors. Connecting these GPUs inside a facility already calls for 1000’s of high-speed ports; extending that cluster to a second website—whereas preserving the deterministic-latency, lossless efficiency of a dwell AI coaching job—requires 1000’s extra coherent optical ports on the scale-across layer. By comparability, conventional DCI sometimes makes use of 1,000 to 2,000 ports to deal with the identical two amenities’ enterprise site visitors. Each use coherent optics over distances exceeding 10 km, and each require sturdy safety. However the scale, site visitors traits, and efficiency tolerances are in a wholly completely different class.
Conventional lossless networks depend on reactive congestion management, which struggles over lengthy fiber distances as a result of the velocity of sunshine means roughly 100 MB of knowledge is in flight on a 100 km hyperlink earlier than stream management may even reply, consuming almost half a contemporary change’s buffer for a single port and precedence. That’s the reason deep-buffered routers, not switches, are the suitable device right here.
AI workloads, nonetheless, supply an vital benefit: they’re predictable sufficient to make proactive congestion management potential, orchestrating site visitors to keep away from congestion earlier than it happens. However hyperlink failures at scale are unpredictable, and once they occur, you additionally want reactive management with deep buffers to soak up the disruption with out forcing the complete job to roll again to a checkpoint and incurring extra expense.
That is the place silicon and coherent optics converge round a single crucial: reliability. On the scale of AI coaching, hyperlink failures are inevitable. A single safety breach or episode of packet loss can erase 1000’s of GPU-hours of labor. Finish-to-end hardware-based safety, deep buffering for failure restoration, and proactive congestion management at the moment are desk stakes. Reliability is prime to Cisco converged AI infrastructure, embedded at each layer.
Energy because the defining constraint and alternative
Energy has develop into the lens by which each architectural resolution in AI networking have to be evaluated.
On the silicon stage, energy effectivity is the deciding issue between a router that’s viable for high-density AI scale-across and one which falls brief.
On the optics stage, the identical logic applies, however the energy problem compounds because the community grows. Pluggable coherent optics scale back energy consumption by eliminating transponders and related consumer optics and permitting direct router-to-router connectivity. Freed-up energy may be redirected to GPUs delivering compute efficiency. However coherent pluggables clear up solely a part of the issue. As scale-across deployments develop from 1000’s of coherent ports to tens of 1000’s, the fiber infrastructure connecting these knowledge facilities should scale in parallel. Extra ports imply extra fiber connections, and extra fiber connections imply extra optical amplification capability alongside these routes. Every of these amplification websites consumes energy of its personal. The result’s a two-sided energy problem: effectivity positive factors inside the information heart on the router-optics interface have to be matched by effectivity positive factors alongside the fiber plant that connects them. Discovering the suitable stability between efficiency and energy at each level within the community is now a first-order engineering downside.
The implication is obvious: scale-across can’t be designed by optimizing silicon and optics independently. They have to be co-designed from the bottom up.
How Cisco is converging silicon and optics in scale-across options
At Cisco, we’ve been constructing towards this convergence for years. The mix of the Cisco Silicon One–powered routing techniques and coherent optics portfolio gives an built-in method designed particularly for what scale-across calls for:
Cisco Silicon One: Cisco Silicon One P200 powers Cisco 8223and techniques to an industry-leading 51.2 Tbps capability, tailor-made for distributed AI workloads. the anticipated forecast progress of Cisco AI orders in fiscal This autumn 2026 to greater than 6 billion. techniques converge routing and switching with spectacular energy effectivity, programmability, and safety, enabling hyperscalers, neoclouds, and sovereign clouds to confidently architect geographically distributed AI environments.
Determine 1.
Coherent modules: Cisco is the coherent market-share chief and pioneer in coherent pluggables. 400G ZR/ZR+ and 800G ZR/ZR+ coherent pluggables are already being deployed in scale-across networks, with over 750,000 400G DSP ports shipped and over 40,000 800G DSP ports shipped. The broad Cisco coherent pluggable portfolio helps the mature requirements outlined by OIF, OpenROADM, and OpenZR+ which have enabled the mass adoption of router-based optics.
Determine 2. Cisco QSFP-DD 800G ZR/ZR+ coherent pluggable
Determine 3. Cisco OSFP 800G ZR/ZR+ coherent pluggable
Open line techniques: Cisco gives two choices for purchasers relying on the use case:
- The brand new Cisco Open Transport 3000 Sequence open line system gives a multi-rail structure that enables a number of fiber pairs to function in parallel so it could actually deal with multi-petabit site visitors over lengthy distances. It additionally helps each C-band and L-band wavelengths, optimizing energy, area, and scalability for scale-across networks.
- The Cisco NCS 1014 metro open line system gives enhanced optical visibility and management that permits coherent pluggable deployments at scale in metro scale-across use circumstances. This contains built-in coherent probe, dynamic achieve equalization, OTDR, and spectral energy monitoring that simplify deploying and working coherent optics which might be disaggregated from line techniques.
Collectively, these capabilities type a scale-across portfolio purpose-built for the reliability, energy effectivity, and scalability that AI infrastructure operators require.
What’s subsequent for scale-across
The scale-across period remains to be early. Networks that can energy the following technology of AI intelligence have to be co-designed, from the coherent DSP and photonic integration on the optical layer, by the silicon and its buffer structure, to the system-level thermal and energy envelope that determines what is definitely deployable at hyperscale.
At Cisco, that’s precisely how we’re approaching scale-across. The Cisco Silicon One adaptive techniques and coherent optics portfolio are designed in shut collaboration internally and with our prospects to meet the particular calls for of scale-across. As AI continues its exponential trajectory, these applied sciences would be the key to unlocking new ranges of intelligence and enabling the following technology of AI infrastructure.
Discover Cisco Silicon One, a scalable and programmable unified networking structure
Further assets
