Winston Churchill once said, “To improve is to change; to be perfect is to change often.” The pace of change in hyperscale cloud data centers continues to rapidly increase due to the insatiable consumer demand for more bandwidth. Given this past year with the COVID-19 pandemic forcing many of us to connect to the outside world primarily through at-home networks, Churchill’s axiom has never been truer.
A recent blog by Dr. Ford Tamer, CEO of Inphi, “Bandwidth in the Age of COVID-19”, details the forces behind the bandwidth growth and the underlying trends driving the data and traffic explosion around the world over the past few years. In this new post, I will explore recent trends happening inside the data center as a result of this bandwidth growth, specifically focused on short-reach physical interfaces at the edge of the network. These interfaces have been served primarily by passive direct-attached copper (DAC) and active optical cables (AOC). However, new solutions are now emerging to support the trend toward faster speeds.
Edge Connectivity in the Data Center Today
Figure 1 provides a high-level representation of a data center network topology with a hierarchical viewpoint of edge connections from the server racks to the top-of-rack (TOR) switches. This edge network sits underneath the higher tier switching infrastructure, providing aggregation and core transport functions.
The edge of the data center is an essential part of the overall cloud networking infrastructure. The use of DAC or AOC technology provides the primary means of connectivity between the server and the TOR switch, or across TOR switches when cascading multiple 1RU switch boxes together into a single aggregated switch. DAC and AOC solutions are also found in high-performance compute (HPC) networks, as well as high-density artificial intelligence (AI) machine learning clusters, with DAC used for the intra-rack connectivity and AOC used for longer distance inter-rack connections.
Passive and Active Cable Solutions
Both passive DAC and AOC connections have been installed in data center networks for multiple generations of equipment cycles. The latest technology is based on a 50Gbps PAM4 electrical host interface, which follows the 200G Ethernet and 400G Ethernet standards (IEEE802.org/3). PAM4 has become the mainstream interconnect solution for the data center and is widely used for electrical and optical signal transmission on 50G, 200G, 400G and forthcoming 800G interfaces. For a closer look at PAM4 and how this signaling technology is transforming next-generation networks, Inphi’s Xi Wang recently published a blog post, “Inphi PAM4 DSP Drives the Future of Inside Data Center, Carrier Network Connectivity, AI, Memory, and Storage.” The post explains what PAM4 is and how it has emerged as mainstream technology for higher-speed signaling.
With host electrical speeds increasing by means of higher bandwidth 12.8Tb and 25.6Tb switches, passive copper cables are running out of steam, given the electrical loss limitations. As a result, new copper cable solutions are being evaluated in the market where electrical PAM4 retimers, or equalizers, are placed inside the cable module to help compensate for the signal impairments and extend the reach and/or reduce the cable thickness of DAC cables. These solutions are commonly referred to as Active Copper Cables (ACC) or Active Electrical Cables (AEC) and are being considered for future edge connectivity, see Figure 2.
As for 200G and 400G AOCs, these products will always incorporate a PAM4 retimer in the module based on link budgets defined by Ethernet standards to drive the target optical reaches.
Figure 3 provides a high-level overview of DAC, ACC and AOC cables, and Table 1 shows a comparison summary of cable solutions addressing 50Gbps PAM4 electrical host signaling for 200G or 400G Ethernet interfaces.
As shown in Table 1, passive DAC technology will always be the most cost-effective power solution versus ACC or AOC. As a result, DAC cables are the preferred choice for short-reach edge connectivity with millions of cables installed in hyperscale data centers today. As electrical line rates increase to 50Gbps PAM4 and higher, new solutions like ACCs or AOCs are being explored to enable both long- and short-reach connectivity.
Edge Connectivity Changing with 100G PAM4
Newly announced 25.6Tb switches are starting to sample into the market with production planned for 2022. To achieve the higher bandwidth and minimize the switch SerDes count, the I/O speed increased from 50Gbps PAM4 to 100Gbps PAM4 on the electrical host interface. The Ethernet Standards body has been working to define the electrical specification on the 100Gbps PAM4 signaling. With this new specification around the corner, cable solutions for edge connectivity are changing to meet the needs of the next-generation hyperscale data center, as shown in Table 2.
As the line speeds increase, the reach with these cable solutions is becoming more challenging compared to solutions with 50Gbps PAM4 signaling where channel losses become more restrictive. The use case for passive DAC is uncertain given the reach and cable thickness required for 100G PAM4 signaling to operate within the rack. The requirement to have a more advanced retimer or equalizer products, leveraging DSP technology to offset signal impairments, is being explored to design next-generation solutions. To support AOC, Ethernet standards are now in process of defining new interfaces based on 100Gbps PAM4 where new 100G VCSEL optical technology is being developed to drive multi-mode fiber in the edge networks.
Inphi PAM4 DSP Retimer Solutions
Consumer demand for more bandwidth does not appear to be slowing down anytime soon, and this ongoing trend is triggering change in the hyperscale data centers for today and tomorrow’s networks. This change is highlighted by the connectivity trends we’re seeing at the edge of the network where short-reach solutions based on more advanced technologies are being developed to satisfy market needs.
As a leader in delivering highly robust PAM4 DSP products to the market, Inphi offers the broadest portfolio of PAM4 DSPs in the industry for both copper and optical cable solutions. Inphi’s PAM4 DSPs are shipping in high-volume production today and have been qualified across multiple hyperscale data center networks, enabling the transition to more bandwidth and higher speeds. For more information on Inphi’s PAM4 DSP solutions, visit www.inphi.com/products/pam-dsp/.