It Isn’t Just Data Centers—AI’s Plumbing Needs an Upgrade

Networking is considered the “plumbing” that moves data and applications inside and between data centers, as well as between data centers and internet-connected devices.

“The amount of traffic that’s starting and will continue to be generated with AI, where you get into a machine-to-machine environment, that amount of traffic is going to be monumental,” said Chris Sharp, chief technology officer of data center operator Digital Realty.

Chip giant Nvidia, networking equipment maker Cisco, data center providers, and internet carriers and exchanges like Lumen Technologies and DE-CIX are eyeing opportunities in a network revamp, which could include gear upgrades, new software tools, and working with network providers to increase capacity and capability.

Cisco last week reported lower quarterly revenue and profit, but noted demand for infrastructure boosted its results.

The $34.61 billion global data center networking market is forecast to reach $118.94 billion by 2033, according to market research firm Straits Research. Sales of data center switches, which route traffic, could nearly double over the next few years, and sales of back-end switches, which connect AI chips, could quadruple, BNP Paribas said.

Global business investment in upgraded AI data center switches alone—which can handle more data than traditional switches—is expected to grow from $127.2 million this year to $1 billion by 2027, according to International Data Corp.

The Teachers Insurance and Annuity Association of America is implementing network upgrades to help become AI-first, said Sastry Durvasula, the money manager’s chief operating, information and digital officer. “The nature of AI workloads demands it,” Durvasula said, “and the competitive landscape necessitates it.”

Training and using generative AI models requires more data movement—known as high bandwidth—at faster speeds—known as lower latency—than other types of internet traffic. “It’s more data than transport networks have ever had to carry in the past,” said Chris Downie, chief executive of data center provider Flexential.

Upgrading a network—or creating a dedicated network—is also required to support large numbers of graphics processing units or GPUs, the power-hungry hardware mainly used to train and run AI. 

The costs are exacerbated because new equipment is pricey. Some AI-ready switches cost at least five times more than traditional data center switches, which can range from a few hundred to a few thousand dollars, according to analyst estimates. Other equipment for upgrades could include routers and software, cybersecurity and automation tools—all of which can be involved in enabling a new network for AI.

Nvidia also makes a networking platform called InfiniBand for moving large amounts of data between Nvidia GPUs inside and between data centers. Ethernet, a competing platform considered less mature for AI networking, is more widely used across all types of data centers and has a larger set of vendors selling equipment to support it, IDC said.

Elon Musk’s xAI recently announced it built a supercomputer “cluster” with 100,000 Nvidia GPUs in Memphis, Tenn., for training and delivering its AI models. The data center housing the supercomputer—both built in just 122 days—uses Spectrum-X, Nvidia’s networking platform based on Ethernet rather than InfiniBand, Nvidia said.

Still, the expected AI networking upgrade isn’t hitting every business at once.

There’s a “long tail” of enterprises still experimenting with AI, said Kevin Wollenweber, a senior vice president and general manager of Cisco’s networking group, and those companies will likely start with the cloud and grow their use of private data centers later.

Many business technology leaders are also training or using AI models in the cloud, rather than in their own data centers. That puts the responsibility on their cloud providers to upgrade network capacity. Many of those providers, including Microsoft and Amazon, have spent billions of dollars to build their own data centers with GPUs and AI-ready networking.

Wayfair relies mostly on Google’s cloud platform, so it doesn’t need to make AI-specific adjustments, said Fiona Tan, the online furniture retailer’s chief technology officer. But if Wayfair’s needs surpass what Google or others can provide, it could explore its own networking options, Tan said.

Kate Johnson, president and chief executive of internet carrier Lumen, said it isn’t surprising big tech companies are building their AI data centers first. Lumen recently struck $5 billion in deals to provide fiber connectivity for Microsoft’s AI data centers. The company also has a deal to provide connectivity for Meta Platforms’s AI infrastructure.

In the next wave, just a few years away, large companies will start building their own private data centers to train and run AI, rather than solely using AI through major cloud platforms, Johnson said.

Some are already there. The digital services and consulting firm Infosys has its own “cluster” of GPUs for building and training small and medium-size AI models, according to Chief Technology Officer Rafee Tarafdar. It also owns GPUs and central processing units, or CPUs, for running AI models. All of that infrastructure, including networking, needs to be regularly upgraded, he said.

Companies serious about implementing AI in their business processes will be serious about their networking requirements, said Naveen Chhabra, a principal analyst at Forrester. “They can clearly see, let’s say three years down the line, that it could be one of their Achilles’ heels if it’s not addressed right at the outset,” he said.

Write to Belle Lin at belle.lin@wsj.com