It comes at you when you least expect it...

Just when it appears that you've got everything figured out, technology has a way of throwing you a curve ball. The curve balls can throw away conventional wisdom built up over years virtually overnight. These changes are many times the actions of a "beast" that has been unleashed unknowingly and will eventually change everything. The beast changed the world of computing and now the question is will a new beast kill traditional core routing approaches and change network architecture?

Let's start with the traditional example – the beat that changed computers. Once upon a time, computers were measured in tons and kilowatts and each design's total production was measured in at most in the tens of thousands of units. Then, the beast was unleashed. At the same time IBM was producing a very successful line of mainframe computers, an almost aside operation created albeit not the first ever personal computer, but the first personal computer that did not sell just in the thousands but sold in the millions. The beast unleashed here was the production of millions of microprocessors. Because tremendous production volume and the demand for more and more performance, this beast rapidly transformed where computing power was found. No longer was it found in the single computer behind the glass window or even in the computer room of a department, this beast was now everywhere. And everywhere meant that economy of scale and the huge revenue opportunity for producing faster and better processors started a real exponential ride.

This beast destroyed empires that could not rapidly adapt. Examples include DEC, Data General, CDC, and many others. Other that could, like IBM, changed both the technology of their mainframes (from ECL to CMOS) and the entire direction of the company.

Today, each of us has more than a supercomputer of the recent past on our desk or laps that cost only a few hundred of dollars. If we now turn our attention to data networking, is there a beast and if yes what and where is it? If we draw a parallel to computing, I contend it is the data networking core. Consisting primarily of large routers, these "mainframe" routers cost hundreds of thousands of dollars – millions when fully populated, just like the old mainframes and moreover, are in production runs of a few thousand units.

It seems pretty clear that Ethernet technology is the beast. Most people are familiar with the Ethernet technology that is used to connect computers to a LAN – and more importantly the fact that you can go to a local office supply or computer store and buy hubs and switches right off the shelf. With 1 Gbps interfaces costing tens of dollars and 10 Gbps interfaces costing around two thousand dollars, this technology beast has dropped the cost of communications equipment dramatically. So if this is the beast, what does is replace? Well, the simple fact is, as I discussed in an earlier post, the cost of an Ethernet port on a carrier-class switch is almost 30-80 times cheaper than the equivalent 10G SONET-based port and almost 10-50 times less expensive than routers with 10G Ethernet interfaces.

So, what does this mean? With the huge increase in bandwidth needs for Internet and enterprise network services, is the current router-based architecture the best way to go? If we draw the conclusion from the computer analogy above, there are hundreds of thousands, if not millions, of high-performance Ethernet switches being sold, as compared to most likely two orders of magnitude fewer large-scale carrier routers. Again, just like in the computer microprocessor example massive production and standards drove increased performance and features that now serve everything from the laptop to the server room (and supercomputers). Will the Ethernet beast run down the same path? Does it really have the right environment to be game changing?

Well, there are several factors. Just like there is competition on price and performance for processors, there is the same competition on the high-volume fundamental chipsets for Ethernet interfaces and switching. Of course, this is not the entire story, but with incredible performance of embedded processors and open source software all the elements are place.

This complete environment of high-performance cheap fundamental switching hardware, high performance processing for higher-level protocols, and open source software for many of the protocols required for an Ethernet switch (and features for routers as well) is now available. Although at one time, open source software was not looked on too warmly by industry, commercial enterprises that have productized open source material and provide traditional software support functions have changed this attitude, and open source is now becoming viewed as being able to support mission critical functions.

The dramatic combination of available hardware and software is the new reality that may have profound and dramatic impact on the current approach to carrier-class MPLS services. Why is this true? In short, it is the need to cost effectively scale. The largest routers today, if we restrict ourselves to a single standard rack, have approximately 160 10G Ethernet ports. The largest and most dense Ethernet switches can be stacked to provide almost 300 or more 10G Ethernet ports in the same space and with much less power consumption (so, as with almost everything, there is a green angle to this as well). Of course, we have to ask the question of whether these switches will have the stability and software features to build carrier-class infrastructure.

The argument that they will is pretty simple. When computer manufacturers believed that microprocessors were toys because of word size or lack or memory management features, and rudimentary operating systems, it did not take very long until microprocessors were the guts of not only laptops and desktops, but mainframe computers as well with all the needed features and more. In the Ethernet world, it is going to be a lot easier to continue to upgrade Ethernet switches with features such as MPLS Fast-ReRoute (FRR) and VRF features for VPNs that will rival and then supplant traditional router approaches. This environment will also will enable new approaches, such as flow-based Layer 2 routing protocols, per flow authorization for security, and others.

In fact, with research projects such as Clean Slate Internet at Stanford University and the National Science Foundation’s Global Environment for Network Innovations (GENI), there are researchers all around the country that are in a position to take these hardware and software tools and create new components and new models for Internet services.

The battle of the LAN was won by Ethernet, the battle of the Edge is being won by Ethernet, so how far away is the battle for the core of the network? In fact it has already begun and for those whose network core router cheese is based on an old-style computer mainframe approach, it may move faster than you think. What will it do to networking equipment manufacturers? And, what will it do to network service providers?

Earth Hour for the Internet

Yesterday was the first Earth Hour day where we are supposed to reduce energy consumption by turning off the lights for one hour in a worldwide energy conservation effort. There has been talk about the Internet being part of the solution for saving the planet (e.g., by enabling telecommuting), but is the infrastructure of the Internet really following "Green" principles?

Let's explore some of the places where energy is used in creating Internet services and get a view about how green they are relative to where they could be. First, there is the optical transport layer. This is the layer that provides both the "wavelengths" that are used to create the backbone of the Internet. It also provides the wavelengths are are used to create optical rings for high-availability SONET-based private lines and Ethernet needed to backhaul customer traffic to Internet edge routers.

Efficient use of power for these systems has always been a design goal and customer criteria. The reason for this is quite simple. Components of these systems (e.g., terminals and optical amplifiers) are located at hundreds of facilities that are in many cases literally in the middle of nowhere (and in some cases, 60 miles from nowhere on the way to nowhere). Controlling power consumption is critical to creating high-availability services, as the power needed by these systems determines the maximum run-time on batteries and generators.

The next place, and the place that today uses more and more power is at the data layer. Whereas optical transport systems may have power consumption of around two to four kilowatts per rack, high-end core routers have power consumptions in the range of 10 kilowatts per rack. Of course, it is not only the power that matters. Virtually every kilowatt of power input to a router becomes heat that must be removed using HVAC systems that use yet more power. Today, the leading router vendors With routers having approximately 100 10Gbps ports per rack yielding about 100 watts per port.

With the rapid growth of the Internet, each additional 10G of cross sectional bandwidth of the network consumes a significant amount of power. Are we doing everything that can be done to reduce the power consumption of the Internet?

First, let's look at the optical transport system. The major power consumption in the optical transport system is the amplification that takes place every 100 kilometers. Technologies exist today that can reduce the number of amplifiers. These include new optical fibers with less attenuation leading to less span loss making is possible to skip existing amplifier locations.

Second, let's examine the core of an Internet backbone network. For most Tier 1 providers is it comprised of approximately 20 core locations throughout the United States. To add 10Gbps of backbone bandwidth across the network, and retaining the typical goal of keeping any route through the network to at most three core routers, you have to add on the order of 100 backbone circuits. This means there are over 200 ports need to be added, or over 20 kilowatts of power (not including cooling requirements). With the explosive growth of Internet traffic (discussed in a previous post), this means that power consumption is growing right along with traffic. With growth today at over 50% compounded annual growth rate in traffic, we are talking about a hot topic.

As with optical transport, there are possibilities in reducing power needed in the Internet core. Most carriers today use full-blown routers (e.g., Cisco CRS-1 and Juniper T-series) to provide backbone MPLS switching and IP routing services. The general reason for this is that these platforms have significant features and proven reliability needed to create a robust and highly-availability network.

The obvious question is whether there is another core architecture that can provide the same backbone capabilities, but do it with less power. The short answer is yes, but the longer answer still requires some additional evaluation. One approach is to use Ethernet switching instead of router-based MPLS. From a power perspective, some of these devices use less than 20 watts per 10G port, or approximately 80% less electricity than a full-blown router.

However, there is no free lunch. There are reasons that high-performance routers have been used instead of Ethernet switches. These include technical features, operational issues, and robustness. Technical features include limitations on complex access control lists and rate limiting, which are tools that are commonly used to provide protection of network element control planes. Operational issues include the lack of comprehensive Ethernet OAM tools, making it difficult to perform fault detection and isolation, and to identify the root cause of poor performance. Finally, using Ethernet switches still requires backbone protection mechanisms that ensure high-availability backbone services. Today, much of this is done via MPLS Fast Re-Route, and there are Ethernet switches that provide this capability. There are other protection mechanisms, but they are either not robust enough or the technique is not proven on a nationwide scale. Other important features, such as hitless software and hardware upgrades, need improvement.

Finally, Business week in its March 20, 2008 magazine has a detailed article (also commented on by Bill St. Arnaud) about the issue of powering and cooling the data centers. It is these data centers where the applications that we know and love, such as Ebay, Google, YouTube, Yahoo, and others find there life. Finding "Green" locations, such as locations like Iceland with Geothermal power, and technologies to reduce power consumption is clearly on the minds of corporate executives eager to both reduce costs and make a little positive PR at the same time.

Apparently evident in the data center business, perhaps the most important question is whether there is an economic advantage for the major Internet providers to move towards a more power efficient transport of IP packets. There is always a significant amount of organizational and technical inertia that keeps network providers from radically changing their approach. However, with the cost of energy increasing and the rapid growth in Internet demand, the need for additional capital investment needed to keep pace may open up a significant opportunity to move towards both greener technology and greener architectures.