Subscribe to Blog Notification Emails

Latest Blog Post

Which Technology Will Prevail for Cloud Networking?

Jayshree Ullal
by Jayshree Ullal on Apr 13, 2009 12:06:56 AM

As we enter this new era of dense servers and multi-core cloud computing, one must pay closer attention to the choice of the right cloud networking technology for seamless compute, network and storage access. I find myself in the midst of "Ethernot vs. Ethernet" choices. Three candidates present themselves as viable options: 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), and InfiniBand. All three technologies boast of high speed at 10 Gbit/s or higher, and promise to virtualize I/O by having other network protocols run inside their fat pipes. Each one comes with their nuances. Let's quickly review the choices.

10 Gigabit Ethernet is clearly the most widespread and standards-based technology. It is a natural progression from 1 Gigabit Ethernet for servers and storage access, and relies on widely deployed Layer 2 and Layer 3 protocols. 10GbE provides cost effective server connectivity with low and predictable latency of a few microseconds with adequate buffering. Non-Blocking Cloud Networking topologies can be constructed with active uplinks multipathed (at L2 or L3) in a two-tier construct spanning 100-1000+ nodes. Seamless storage access is possible through iSCSI-based block storage or network attached file storage using 10GbE networking. Standards based products such as Arista's 71XX family are only 2-3 times the price of 1GbE, with prices as low as $500/switch port. Smooth transitions and familiar operations are encouraged by imminent 10GbE LAN-on-motherboard server designs and the emergence of 10GBASE-T, which allows 10GbE to run distances of 30-100 meters over existing Category 5 or new Cat 6 cabling.

While iSCSI and File Storage can run on tried and true 10GbE, the SAN Fibre Channel world awaits Fibre Channel over Ethernet (FCoE) only if convergence of SAN and Ethernet is desired. Taking Fibre Channel protocols and layering them on Ethernet requires special changes to standard Ethernet, such as priority flow control (PFC). Existing Ethernet is inherently defined for graceful drop of packets, which will break Fibre Channel links, so servers require special NICs with FCoE support in order to link to this emerging infrastructure. Standards for FCoE and and associated PFC are being finalized in 2009 by the IEEE Data Center Bridging (DCB) group, and standards-compliant interoperable products are expected in the 2010 timeframe. Deployment of new storage protocols like FCoE will take time. The convergence of storage protocols such as ISCSI (and future FCoE) can be viewed as an optional overlay upon 10GbE transport, rather than a separate flavor.

InfiniBand (IB) is already deployed in large high-end HPC clusters. InfiniBand requires special host adapters and switches, and often a significant number of gateways to connect the IB island to the Ethernet galaxy. It has been installed at 8 Gbit/s and double data rates of 16 Gbit/s, and is now transitioning to quad data rates of 32 Gbit/s. InfiniBand boasts of the lowest latencies, required in ultra high performance compute clusters, and will remain so in the HPC niche. It is important to note that InfiniBand is a good server-interconnect, but it does not provide the complete flow control for scalable networks, needs gateways, and requires a dedicated operations team. In my long career in networking, I have learned not to bet against Ethernet. Despite having been in the midst of logical reasons on the deterministic virtues of Tokens with 100 Mbps FDDI, or 53 byte cells of ATM, these technologies become a transient and niche chapter in history. The mainstream choice and winner is obvious. Yes, it is 10 Gigabit Ethernet. FCoE may become an overlay option over time when standards and interoperability mature. The introduction of 10GBASE-T combined with higher speed evolutions to 40 gigabit and 100 gigabit Ethernet, make 10GbE migrations more compelling. Welcome to the new, yet familiar world of 10GbE based cloud networking!

Your feedback is welcome at feedback@arista.com
References:
Technology Partner Solutions
IBRIX Cloud Storage
The Widespread role of 10GigE in Storage

Opinions expressed here are the personal opinions of the original authors, not of Arista Networks. The content is provided for informational purposes only and is not meant to be an endorsement or representation by Arista Networks or any other party.
Jayshree Ullal
Written by Jayshree Ullal
Jayshree Ullal is a networking executive veteran with 30+ years of experience. In 2018 Barron's named her one of the "World's Best CEOs." In 2015 she was co-awarded "EY 2015 Entrepreneur of the Year" across National USA and "#3 IT Industry Disrupter" by CRN. In 2005, she was named one of the "50 Most Powerful People" by Network World and one of the "Top Executives" by Forbes magazine 2012. As President and CEO for a decade, Jayshree led Arista Networks to a successful IPO in June 2014 at NYSE. She is responsible for building a multibillion dollar business in cloud networking and has forged strategic alliances with Microsoft, HP and VMware to name a few.

Related posts

Tech Time is Real Time

Silicon Valley is both an addiction and passion where entrepreneurs seek the realm of the impossible. Real-time language...

Jayshree Ullal
By Jayshree Ullal - December 11, 2018
Cognitive Campus - Next Frontier

Arista’s focus on disruption, with datacenters and routing, transforming siloed places in the network to seamless Places In the...

Jayshree Ullal
By Jayshree Ullal - October 16, 2018
Reflections on the Cloud Networking Decade

When I joined Arista ten years ago, we were in the midst of developing a novel purpose-built software architecture from a clean...

Jayshree Ullal
By Jayshree Ullal - September 17, 2018