Subscribe to Blog Notification Emails

Latest Blog Post

The Symmetry between Gigahertz and Gigabits in Cloud Networking

Jayshree Ullal
by Jayshree Ullal on Jul 14, 2009 3:30:33 AM

One of Om Malik's recent blogs I read in Gigaom was thought provoking (as usual!). It made me realize that we have reached a tipping point with the introduction and deployment of powerful multi-core processors. They have begun to challenge the foundation of today's anemic network connectivity. A network running at 1 gigabit is becoming the weak link in the data center. The adoption of just about every web scale application be it File Systems protocols, Map-Reduce frameworks with Hadoop clusters or Mem-cached's giant hash tables, are placing far greater demands on scalability. We are no more limited by compute power as dense cloud computing enables processing speeds of 1.6 peta-flops, which even out paces Moore's law. Compute capacity could vastly outstrip network bandwidth by 2:1 if we are not prepared. In fact, we are more network-bound by Nielsen's Law.

Consider a simple rule of thumb to guide compute-network ratios.

1 gigahertz of CPU = 1 gigabit per I/O.

This means a single core 2.5 gigahertz X86 CPU already requires 2.5 gigabits of networking, and a quad-core needs four times as much, that is 10 gigabit networking. In Arista's customer applications we have seen peak traffic loads of 10Gbps of traffic over Ethernet through a single Xeon 5500 based server. The intensity of workloads and compute utilization can vary from large data transfers to clusters, and file storage. What is clear is that networking throughput is no more as compute-bound, but limited by the fine-tuning of the stack and the achievable network performance. High Performance Compute (HPC) workloads are likely to peak more than 10Gbps per server especially when you couple it with storage traffic on the same network.

The Brownian Effect of Virtualization:

The exponential effect of virtualized servers creates thousands of virtual machines running around in Brownian motion saturating the network I/O. In a typical virtualization deployment, it is quite common to see servers running VMware’s ESX4.0 and deploying an average of six 1-Gigabit Ethernet NICS (Network Interface Cards) to cope with the multiplicity of servers. Another test came close to saturating even a 10GbE link between a pair of octal core Xeon 5500 machines each running four Xen VMS and NICS with I/O Virtualization features. By replacing multiple 1Gigabit with a single 10Gbps Ethernet pipe (NIC and Switch capable of virtual portioning) one can simplify, save costs and increase throughput in real production environments. More on this virtualization topic in my next blogs.

Two Tiered Cloud Leaf-Spine Designs:

As virtualized servers push the limits and drive 10Gbps of Ethernet traffic in access or "leaf" of the network, we must pay equal attention to the ripple effect in the core backbone or "spine" network (Refer to my Jan Blog). Take a configuration with racks of 40 servers each, putting out a nominal 6-9 Gbps rate of traffic. Each rack will have a flow of 240-360 Gbps and a row of 10-40 racks will need to handle 2 Tbps to 10+ Tbps. Especially when flows between servers are in close proximity to each other, two - tiered non-blocking leaf-spine Arista 71XX based-architectures are optimal.

As multi-cores and virtualization are deployed widely, a new scale of bandwidth is needed for 10GE/40GE and eventually 100GE. Otherwise the network becomes the limiting factor for application and cloud-based performance and delivery. This translates to multiple 1000-10,000 node clusters and terabits of scale! Welcome to the new era of cloud networking! Your thoughts are always welcome at feedback@arista.com

References:
Nielsen's Law

Opinions expressed here are the personal opinions of the original authors, not of Arista Networks. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Arista Networks or any other party.
Jayshree Ullal
Written by Jayshree Ullal
Jayshree Ullal is a networking executive veteran with 30+ years of experience. In 2018 Barron's named her one of the "World's Best CEOs." In 2015 she was co-awarded "EY 2015 Entrepreneur of the Year" across National USA and "#3 IT Industry Disrupter" by CRN. In 2005, she was named one of the "50 Most Powerful People" by Network World and one of the "Top Executives" by Forbes magazine 2012. As President and CEO for a decade, Jayshree led Arista Networks to a successful IPO in June 2014 at NYSE. She is responsible for building a multibillion dollar business in cloud networking and has forged strategic alliances with Microsoft, HP and VMware to name a few.

Related posts

Tech Time is Real Time

Silicon Valley is both an addiction and passion where entrepreneurs seek the realm of the impossible. Real-time language...

Jayshree Ullal
By Jayshree Ullal - December 11, 2018
Cognitive Campus - Next Frontier

Arista’s focus on disruption, with datacenters and routing, transforming siloed places in the network to seamless Places In the...

Jayshree Ullal
By Jayshree Ullal - October 16, 2018
Reflections on the Cloud Networking Decade

When I joined Arista ten years ago, we were in the midst of developing a novel purpose-built software architecture from a clean...

Jayshree Ullal
By Jayshree Ullal - September 17, 2018