HomeAbout Me

Cisco is using Linux virtualization and 40 core CPUs for its next generation routers

By Colin McNamara
March 10, 2008
6 min read
Cisco is using Linux virtualization and 40 core CPUs for its next generation routers

Cisco recently released a new series of router called the Aggregation Services Router, or ASR for short. This series of routers is mainly targeted at the service provider market, where it is targeted as a single chassis solution for what is called the “triple play” - Voice, Video, and Data. More accurately it can be targeted to the new “quadruple play” of Voice, Video, Data and Security. The ASR1000 accomplishes this by leveraging two key technologies. These are a new operating system, IOS-XE which uses the Linux kernel as its foundation, and Cisco’s new QuantumFlow 40 core processor.

IOS-XE takes the best elements out of Internet Operating System (IOS) which has its roots in a closet at Stanford, and combines them with the most successful open source technology ever - Linux. Cisco is leveraging Linux virtualization technologies such as Kernel Based Virtual Machines to protect against operating system failures as well as to allow for In Service Software Upgrades (ISSU).

To really appreciate this, we first have to dive down into the overall architectural changes of the ASR1000. The largest change that Cisco has made was to implement separate forwarding and control planes. In the past, Cisco routers would have the processes responsible for forwarding traffic, and the processes responsible for configuring the router running on the same root operating system. The side effect of this is that if you want to upgrade the root operating system of your router, you are going to have to interrupt the traffic flowing through it to do so, or have a physically separate route processor to take over while you rebooted. This is a big headache operationally, and effectively forced engineers to design in separate physical chassis to meet high uptime requirements.

What Cisco has done to address this, was to mirror changes made in their storage and carrier routing portfolios. Both of those product lines utilize the operating system to push commands into advanced processors that exist on the line cards themselves. The ASICs on the line cards are designed to work in a distributed fashion, so that production traffic never goes up into the router processor (or sup engine). This in effect ensures that the control and forwarding planes can exist as independent elements.

If you look at the architecture, you will notice 3 main zones. The upper zone is what we would normally describe as the control plane. This is where the higher level functions such as your routing processes, ssh daemons, snmp daemons, and shells live. In short, if you configure or read something, you are going to do it here. The only time traffic flows through this plane is when you are doing a thing called process switching. Keep in mind this is a rare occurrence and usually occurs because of an oversight in your network designs.

By separating the control and forwarding planes, this allows Cisco to basically run a management station on the router, that programs chip sets in the line cards on the fly. This in my opinion is where the true power of this architecture comes through. By separating the two functions the software engineers are free to utilize powerful open source technologies such as Kernel-based Virtual Machines, and the Linux kernel, while letting the integrated circuit engineers design blazing fast chips which allow full functionality at line rate.

What benefits should we receive from a virtualized control plane? First, in larger routing and switching chassis (including the top end of the ASR1000 line) you normally have physically redundant route processors (RP)/ supervisory engines(SUP). The operating systems on these RP’s synchronize many things, including configuration, process state, routing tables, security associations and much more. The primary reason for this is if you have a failure in the active RP, you can failover to the standby RP without interrupting traffic flows. They also can be used to streamline the software upgrade process by only upgrading one RP at a time, and then gracefully transferring traffic to it. Once proper operation is verified, the backup RP can be brought up to the same code revision. In any production environment this is highly desirable, and helps immensely in the battle for five nines.

The ASR1000 takes the redundant RP concept seen in high end chassis, and allows you to implement redundant upgrades, as well as protection against software failure, with only one physical route processor. This is done by utilizing Linux kernel virtualization. Instead of running the control plane directly on the production hardware, a small kernel is inserted. Booting from that are two copies of IOS-XE. These run independently, and synchronize state and configurations just as if you had two physically separate route processors. What this means in operational English, is that where in the past, you would have to either have two devices, or a larger device with redundant RP’s to upgrade without disruption, you can now have that same ease of maintenance, in a much smaller (and at the end of the day, less total cost) package.

Below this is the forwarding plane. It plugs into a high speed interconnected fabric which all line cards and RP’s are redundantly connected to. Items in this plane include buffer memory, Cisco Express Forwarding (CEF) ASICs, and now the new QuantumFlow processor. This is normally where you would find your DCEF enabled line cards, fibre channel and Nexus7000 line cards, as well as the modules for the ASR1000 routers. When properly utilized, traffic should be relatively isolated to this tier, and function independently from the control plane.

The shining star of the ASR1000’s forwarding plane is a group of chips that is referred to as QuantumFlow. The QuantumFlow architecture itself merges Cisco’s strength in integrated circuit design, with its strengths in IOS software design. In the past, Cisco would design ASICs for specific functions, and then write commands down into them. This has worked very well, until the point that a new feature came out that couldn’t leverage the fixed configuration of an older ASIC. Your choice at that point was generally to process switch for that feature (which is slower, and honestly bad form), or upgrade your cards to the newer ASIC design. The QuantumFlow chipset approaches this problem from a new angle. The first chip in the set (Popeye) is designed to be field programmable in C, as well as no fixed internal pipelines. This combined with utilizing 40 cores running between 900 and 1200 megahertz allows the programmers to utilize parallel processing techniques to utilize an immense amount of processing power in real time.

To put things into perspective, remember when you got your first multi-core laptop or desktop. You were able to say watch a DVD, as well as compile code at the same time, while continuing to have a responsive workstation. Now imagine what you could do with a 40 core processor. This is the kind of power that we are talking about. Now imagine, that not only is your workstation immensely powerful, but you could also offload common jobs such as running daily builds, or encoding videos to another machine (or in this case processor).

In the ASR1000 this processor is called Spinach. And of course just like the cartoon, Popeye’s potential really comes to light when combined with Spinach. Spinach is a separate chip, that is used as a traffic manager. This chip handles queueing and quality of service, ensuring that the proper packets arrive at the proper time, as well as interconnecting with cryptographic offload engines so it can equally apply services to encrypted flows.

At the end of the day, the most important question is not how fast something is, or how cool it is. The question is what can it do for me? By leveraging this new architecture the ASR1000 is now able to do line rate inspection of traffic using Network Based Application Recognition (NBAR), Support 128,000 queues for deep quality of service, secure and encrypt data using zone based firewalls and embedded crypto engines, segregate traffic using MPLS, integrate advanced voice and video functionality, as well as providing full Netflow v9 support for all of the above. It provides all of these services in an always on solution utilizing Linux virtualization, as well as leveraging a flexible chip set architecture that allows for field programmable improvements in the future.

My hope is that after reading this article that you are in a better position to understand how Cisco is leveraging open source technology and integrated circuit designs to improve the foundation of the internet. In upcoming articles I will be discussing design scenarios utilizing these features in this product, as well as highlighting other areas where Cisco is embracing both open source technology, as well as open architectures that can properly leverage projects such as Linux, Ntop, Wireshark and more.

Learn more:

  • Linux Kernel-based Virtual Machines
  • Cisco’s ASR1000
  • Cisco QuantumFlow

Tags

cisconetworkingvirtualizationlinuxrouter-architecture

Share

Previous Article
Reader question - Why are corporations looking for BGP experience?
Colin McNamara

Colin McNamara

AI Innovation Leader & Supply Chain Technologist

Topics

Business & Strategy
Personal & Lifestyle
Planet & Purpose
Technology & Innovation

Related Posts

Open Letter to Cisco: Improving the DevNet Developer Experience
February 11, 2014
7 min
© 2025, All Rights Reserved.

Quick Links

About MeContact Me

Social Media