OpenFlow/SDN is Not a Silver Bullet for Network Scalability
Ivan Pepelnjak (CCIE#1354 Emeritus) is Chief Technology Advisor at NIL Data Communications, author of numerous webinars and advanced networking books, and a prolific blogger. He’s focusing on data center and cloud networking, network virtualization, and scalable application design.
OpenFlow is an interesting emerging networking technology appearing seemingly out of nowhere with much hype and fanfare in March 2011. More than a year later, there are two commercial products based on OpenFlow (NEC’s Programmable Flow and Nicira’s Network Virtualization Platform) and probably less than a dozen production-grade implementations (including Google’s G-Scale network and Indiana University’s campus network). Is this an expected result for an emerging technology or another case of overhyped technology hitting limits imposed by reality?
OpenFlow-based solutions have to overcome numerous problems every emerging technology is facing, in OpenFlow’s case ranging from compatibility with existing chipsets to incomplete and fast-changing specifications (and related compatibility issues), but they’re also hitting some hard scalability limits that we’ll explore in the rest of this article.
What Is OpenFlow?
In case you haven’t been exposed to OpenFlow, here’s a 30-second introduction. Every networking device has a forwarding (or data) plane, where the stuff doing the real work (be it hardware or software) uses optimized data structures (forwarding tables or forwarding information base) to forward packets, sometimes munging them in transit (ex: Network Address Translation and load balancers).
The forwarding tables are set up by software – the control plane – that is usually embedded in the same networking device. There are a few notable exceptions, including wireless access points with central controllers and data center switch clusters using Borg architecture (no, that’s not an official term for a cluster of devices with a single control plane).
OpenFlow is a standardized protocol that can be used between control- and forwarding planes. The architecture where the control plane is centralized and separate from the physical devices is called Software Defined Networking according to the definition promoted by the Open Networking Foundation.
Using OpenFlow, you could buy forwarding devices from any vendor (or multiple vendors) and program their forwarding tables with open-source control plane software running on low-cost commodity x86 hardware ... or so the theory goes.
For a more in-depth introduction to OpenFlow, watch the Networking Devices Planes of Operation and OpenFlow-Based Forwarding videos or the recording of OpenFlow and Softwre Defined Networking webinar presented by Greg Ferro. Lots of OpenFlow-related material is also available on Open Networking Summit web site.
Will OpenFlow Scale?
This is probably the most relevant question for the readers of this blog ... and it’s a wrong question. OpenFlow is just a protocol like HTTP, and doesn’t make much sense to talk about scalability of a client-server protocol.
We all know early versions of HTTP contained severe scalability issues (lack of pipelining, for example), and there are probably one or two glitches in OpenFlow, but the protocol itself seems solid, and I’m positive the next versions (we started with OpenFlow 1.0 last March and the 1.3 release has just been ratified) will improve it further.
A more appropriate question is “Will the Software Defined Networking scale?”, and the answer is (like with web-based applications) “It depends on the architecture and implementation details.”
Scalable SDN Example
Some solutions use OpenFlow to program a number of independent devices that don’t interact with each other. You could compare these use cases with a scale-out application with no shared state (networking devices) and a back-end database (OpenFlow controller). As we all know, such application architectures have few scalability challenges; their scalability depends primarily on the back-end database. The situation is no different in the SDN world; let me give you two real-life examples.
Nicira’s Network Virtualization Platform (NVP) implements scalable virtual networks used primarily by IaaS cloud providers. NVP relies on OpenFlow to program the virtual switches in Linux-based hypervisors (Xen or KVM, for example). Each hypervisor switch is totally independent from all other switches and does not interact with them directly; user (VM) traffic is encapsulated in IP datagrams and sent to the network, which is supposed to provide end-to-end IP transport. NVP is focused exclusively on the hypervisor switches and does not interact with the physical networking devices at all; even server NIC bonding and failover is left to the Linux kernel.
NVP’s scalability is thus limited exclusively by the scalability of the controller application. As one would expect these days, NVP uses Paxos and a scale-out back-end database that stores virtual network and VM information.
Indiana University’s campus deployment is planning to use a very similar architecture. They plan to use OpenFlow to implement user access control on edge devices. Each edge device would be controlled independently. The edge devices usually don’t interact with each other (they are usually connected to aggregation or core switches), and the amount of changes generated by each edge device is directly proportional to user churn (= low), so the amount of OpenFlow processing their solution would have to support would have a well-known upper bound. In their planned solution, they would use OpenFlow to intercept initial user traffic (until the user passes authentication) and to download access policy (usually in form of Access Control List – ACL) to the edge device (wireless or Ethernet switch).
The only shared object in the system Indiana University is planning to build is the central policy database, which would be accessed by a cluster of OpenFlow controllers interacting with individual networking devices. Scaling the controller cluster to cope with the growing number of controlled devices is thus a trivial exercise. For more information on their current and planned solution, please view the presentation Matt Davy gave at NANOG 50; he also had a session at Interop 2012 and Open Networking Summit 2012.
Using SDN/OpenFlow in Network Core
Some proponents of OpenFlow and SDN focused on a much tougher task: reinventing the network core. Some ideas are obviously several orders of magnitude beyond being implementable in a realistic world – a prime example would be global load balancing with per-session flows installed in every Internet router; you simply cannot download hundreds of thousands of flows into a single switch or router.
Other approaches seem more realistic. For example, NEC has implemented an OpenFlow controller that is supposed to control the whole data center network. But even there, their implementation quickly hit real-life limits:
- The number of flows a physical device can handle in hardware is limited, and the software-based devices are still too slow. It’s thus impossible to implement very granular end-to-end traffic control.
- A device using hardware-based packet forwarding can install only a limited number of flows in a time period (usually less than thousand flows per second), a fact that is well known to manufacturers and users of high-end routers.
The above limitations could be solved with the next-generation chipsets; after all, modern layer-2 switches perform MAC address learning in hardware, and there’s no reason OpenFlow-optimized chipsets couldn’t do something similar.
However, it turns out large-scale networks with distributed intelligence (control plane) perform inherently better than systems with centralized control – the reason IP routers with distributed routing protocols became so prevalent in the last two decades that we barely remember what SONET/SDH, Frame Relay or ATM were (hint: all three technologies relied on centralized virtual circuit setup).
It’s simply impossible to recover from a node or link failure in 50 milliseconds (a typical requirement in networks handling voice traffic) when it takes longer to get a reply from the central controller. There’s also the “slight” problem of network devices losing connectivity with the central controller if the primary uplink fails.
Offloading some of the intelligence and/or installing precomputed alternate paths into the network nodes can solve both problems. Traditional WAN technologies (example: SONET or SDH) relied on redundant circuits to implement fast failover times (because the controllers couldn’t be relied upon to find and install an alternate path in time), in many cases wasting half of the bandwidth – another reason why service providers started preferring IP+MPLS-based networks over traditional optical networks.
Google was quick to recognize the challenges – they use OpenFlow in their G-scale network, but only within a data center, where a cluster of OpenFlow controllers manages local devices. They use traditional routing protocols (BGP+IS-IS) between sites and further influence traffic flow with proprietary traffic engineering technology similar (in functionality) to MPLS-TP. For more details, watch Open Networking Summit 2012 presentations from Urs Hoelzle and Amin Vahdat, or read my analysis of what they shared with us.
NEC also hit limits of real-time control with their ProgrammableFlow product. A single OpenFlow controller can control only a few dozens of top-of-rack switches without supporting “linecard protocols” like LACP or BFD, or running routing protocols or spanning tree protocol (STP) with the external devices. A network built with their OpenFlow controller interacts with the outside world through static routes and static link aggregation groups (LAG).
Conclusions
Every new technology triggers an unlimited wave of enthusiasm and OpenFlow is no exception. Development teams with long networking experience have quickly realized that OpenFlow or SDN cannot overcome limitations inherent in asynchronous distributed systems with unreliable communication paths. Some of them focused on controlling a large number of independent edge devices, others decided to use the advantages of OpenFlow while retaining the distributed nature of the system that gives large-scale IP networks (example: the Internet) their resilience.
Google’s G-scale network is a perfect example of optimal SDN architecture in a large-scale WAN environment. They use OpenFlow solely to control of a cluster of local devices and combine local controllers with centralized path computation and traditional time-proven technologies (routing protocols).
We can thus expect major OpenFlow/SDN-based advances in the network edge, where individual devices don’t interact with each other, and minor impact of OpenFlow in the network core … unless, of course, we’re talking about Google-scale networks, where every bit of flexibility and increased WAN utilization counts.
