Going Horizontal on the Vertical

In this case, Going Horizontal on the Vertical is not a catch phase from the original Outer Limits television series where the announcer calmly proclaims that “we control the horizontal and we control the vertical”. The point of this post is to get the point across that almost no matter how diligent telecommunications carriers think they are when they develop products, or when customers write statement of objectives, invariably the result is a set of vertical services or sets of requirements that leaves the ultimate reason for the services in the first place out of the equation – the mission of the organization.

Why are we in this situation? Isn’t it clear that the purpose of network services is to provide an infrastructure and Information Technology (IT) resources to support the mission of the organization, company, or government agency?

Our first look will be from the perspective of a telecommunications service provider. Historically, and fundamentally, there are two sets of organizations that play a role in the development of the services offered. These are the Product Management organization, and the Technology Management organization. Note that neither of these organization’s names have anything to do with a customer focused solution. They are focused on an individual product and a set of technologies to enable that product. It actually goes further south from there – and although some aspects of this have improved over time, an example from my direct professional experience will shed some light on just how bad thing were.

In the beginning, technology created Asynchronous Transfer Mode (ATM) and Frame Relay (FR) – well not really in the beginning as there as Morse code, Baudot , X.25, Switched 56K, etc, etc… Then, along came the Internet and Internet Protocol Services. Let’s deal with ATM and FR first. So, as a product manager, what was the approach to selling the product? Well, instead of really understanding the problems that customers want to solve by using a network-based data service, you focus on what the box manufactures and standards bodies are telling you. You specify your product not in terms of a set of configurations that can be used to solve a series of customer focused network problems, but in terms absolutely incomprehensibly except by bowing to the alter of some network engineer. You’ve got PVCs, SVCs, DLCIs, VPI/VCIs, CBR, ABR, PCR, SCR, MBR, VBR, VBR-rt, VBR-nrt, UBR, and so on and so on and so on. As sales documentation you take out the glossies from your favorite box vendor and maybe something from an nearly otherwise useless book on ATM and make assertions like: “CBR is great for video, VBR-rt is great for voice, ABR is great for who knows what, UBR is great for pesky email you don’t really want to get, etc”.

Now let’s bring in the customer, who decides that if the carriers are going to declare technical mumbo-jumbo war in describing their products, then they are going to have to get their own technical mumbo-jumbo experts to counter. So, instead of the customer discussing what they are trying to accomplish, they create Request for Proposals that look more like someone picked up the ATM UNI 3.1 specification and picked out every nearly useless and arcane feature and made it a mandatory requirement in the carrier’s response. All this time, the actual mission and reason for the network is nowhere to be seen.

This mismatch between the way that the product is provided and priced to the customer and the real customer need demonstrates that pure technology is not likely to yield customer happiness.

Everyone knew that the reason that people did not like ATM and FR was that there was always a mismatch between the configuration control of the ATM and FR network and the almost exclusively private Internet Protocol (IP) network created at the edge by the customer. So, when public IP services came onto the scene, the Holy Grail was at hand and everything would be happy if you just used the Internet.

So, instead of recognizing that IP services were just another form of data transport, the previous, since time immemorial, approach of creating a new product management organization to care and feed the new IP services product was implemented. And worse, put in opposition to the existing data services product management. This led to the inevitable clash of product managers that arguing that “my product is better than yours because IP is golden and ATM is old junk”.

Because of this clash of religions, each product continued to be managed as independent entities, and each were given network management environments to support the technical features of each product. Even if these were combined into a unified portal, each data product was treaded individually. These management systems re-enforced the technical attributes of the services, but continued to ignore the reality of the customer’s mission. Even worse, service providers that had a managed Hosting service generally had a completely separate hosted services portal, completely focused on the hosting environment and completely oblivious to the network environment that is needed to support complex applications.

In an attempt to remedy part of this issue, the basic approach of carriers was to start developing Managed Network Services. These services were generally based on the carrier creating a Network Operations Center (NOC) that focused on the edge devices of the network. This carrier NOC approach is generally limited to whether the edge routers of a customer are working as designed and to providing a single point-of-contact for trouble management of the underlying carrier-based data transport network. In some cases, these NOCs may use the existing vertical service portals (e.g., to help determine performance issues with an MPLS VPN), or receive a data feed (e.g., in XML format) of the actual data elements that describe the performance of the underlying transport network. However, as previously mentioned, this is a limited view of the enterprise, as the network may be working, but some critical application is failing.

In some cases, this may not be a significant issue, as the operation that is watching a Web-server will address the server failing and is generally independent of a network service. The increase in the richness (i.e., voice and video) in Web services means that this independence between network performance and Web-server operation is disappearing and calls out for a unified environment so that the impact of network performance on application performance is clear and well reported.

How do we get horizontal on the vertical? There are two cases, one for carrier provided Network Management services and for the case where this function is either in-house to the enterprise or outsourced to a third party.

Carriers must start recognizing that they are actually application enablers and that their products are not the application in themselves. This means that all the vertical tools that are created to monitor and report on the technical aspects of a service (e.g., latency, jitter, packet loss, QoS operation, etc.) are actually components in the proper operation of an end-to-end application. Providing this data in Web page is a start, but these services need to have an Application Programmer Interface (API) that enables the performance parameters to be used to create the “big picture” of the status of an application. This API can be used both by the carrier in an integrated environment that combines multiple services, for example network and hosting, into an application view for their customers. In addition, this API can be used to provide a third party the information necessary to customize a network management platform that integrates information from various sources (i.e., internal to the enterprise and from service providers) to provide the application situational awareness needed to effectively operate the enterprise.

Another aspect that is generally not integrated into these systems so called Change Management process which encompasses items such as service affecting network upgrades. Currently, this is yet another process for each product and is generally not integrated either into the vertical management portal for individual services or into consistent Change Management portal for enterprise customers.

Carriers have the ability to reach into their systems for network data, security services data, trouble management information, change management information, and managed applications performance data. They are uniquely positioned to be able to take the high-ground on developing Web portals that can be rapidly customized using the data feeds from the multiple services being provided. This can create real application performance awareness to the enterprise and incorporate the information and visibility necessary to understand the impact of each element of the services being provided on the applications and enable more rapid root cause analysis and therefore more rapid trouble management response and repair.

Getting Horizontal on the Vertical really means understanding customers better and the inter-related nature of their use of services and applications. In my view, this is the ultimate Vertical penetration you can achieve in any customer’s operation.

It comes at you when you least expect it...

Just when it appears that you've got everything figured out, technology has a way of throwing you a curve ball. The curve balls can throw away conventional wisdom built up over years virtually overnight. These changes are many times the actions of a "beast" that has been unleashed unknowingly and will eventually change everything. The beast changed the world of computing and now the question is will a new beast kill traditional core routing approaches and change network architecture?

Let's start with the traditional example – the beat that changed computers. Once upon a time, computers were measured in tons and kilowatts and each design's total production was measured in at most in the tens of thousands of units. Then, the beast was unleashed. At the same time IBM was producing a very successful line of mainframe computers, an almost aside operation created albeit not the first ever personal computer, but the first personal computer that did not sell just in the thousands but sold in the millions. The beast unleashed here was the production of millions of microprocessors. Because tremendous production volume and the demand for more and more performance, this beast rapidly transformed where computing power was found. No longer was it found in the single computer behind the glass window or even in the computer room of a department, this beast was now everywhere. And everywhere meant that economy of scale and the huge revenue opportunity for producing faster and better processors started a real exponential ride.

This beast destroyed empires that could not rapidly adapt. Examples include DEC, Data General, CDC, and many others. Other that could, like IBM, changed both the technology of their mainframes (from ECL to CMOS) and the entire direction of the company.

Today, each of us has more than a supercomputer of the recent past on our desk or laps that cost only a few hundred of dollars. If we now turn our attention to data networking, is there a beast and if yes what and where is it? If we draw a parallel to computing, I contend it is the data networking core. Consisting primarily of large routers, these "mainframe" routers cost hundreds of thousands of dollars – millions when fully populated, just like the old mainframes and moreover, are in production runs of a few thousand units.

It seems pretty clear that Ethernet technology is the beast. Most people are familiar with the Ethernet technology that is used to connect computers to a LAN – and more importantly the fact that you can go to a local office supply or computer store and buy hubs and switches right off the shelf. With 1 Gbps interfaces costing tens of dollars and 10 Gbps interfaces costing around two thousand dollars, this technology beast has dropped the cost of communications equipment dramatically. So if this is the beast, what does is replace? Well, the simple fact is, as I discussed in an earlier post, the cost of an Ethernet port on a carrier-class switch is almost 30-80 times cheaper than the equivalent 10G SONET-based port and almost 10-50 times less expensive than routers with 10G Ethernet interfaces.

So, what does this mean? With the huge increase in bandwidth needs for Internet and enterprise network services, is the current router-based architecture the best way to go? If we draw the conclusion from the computer analogy above, there are hundreds of thousands, if not millions, of high-performance Ethernet switches being sold, as compared to most likely two orders of magnitude fewer large-scale carrier routers. Again, just like in the computer microprocessor example massive production and standards drove increased performance and features that now serve everything from the laptop to the server room (and supercomputers). Will the Ethernet beast run down the same path? Does it really have the right environment to be game changing?

Well, there are several factors. Just like there is competition on price and performance for processors, there is the same competition on the high-volume fundamental chipsets for Ethernet interfaces and switching. Of course, this is not the entire story, but with incredible performance of embedded processors and open source software all the elements are place.

This complete environment of high-performance cheap fundamental switching hardware, high performance processing for higher-level protocols, and open source software for many of the protocols required for an Ethernet switch (and features for routers as well) is now available. Although at one time, open source software was not looked on too warmly by industry, commercial enterprises that have productized open source material and provide traditional software support functions have changed this attitude, and open source is now becoming viewed as being able to support mission critical functions.

The dramatic combination of available hardware and software is the new reality that may have profound and dramatic impact on the current approach to carrier-class MPLS services. Why is this true? In short, it is the need to cost effectively scale. The largest routers today, if we restrict ourselves to a single standard rack, have approximately 160 10G Ethernet ports. The largest and most dense Ethernet switches can be stacked to provide almost 300 or more 10G Ethernet ports in the same space and with much less power consumption (so, as with almost everything, there is a green angle to this as well). Of course, we have to ask the question of whether these switches will have the stability and software features to build carrier-class infrastructure.

The argument that they will is pretty simple. When computer manufacturers believed that microprocessors were toys because of word size or lack or memory management features, and rudimentary operating systems, it did not take very long until microprocessors were the guts of not only laptops and desktops, but mainframe computers as well with all the needed features and more. In the Ethernet world, it is going to be a lot easier to continue to upgrade Ethernet switches with features such as MPLS Fast-ReRoute (FRR) and VRF features for VPNs that will rival and then supplant traditional router approaches. This environment will also will enable new approaches, such as flow-based Layer 2 routing protocols, per flow authorization for security, and others.

In fact, with research projects such as Clean Slate Internet at Stanford University and the National Science Foundation’s Global Environment for Network Innovations (GENI), there are researchers all around the country that are in a position to take these hardware and software tools and create new components and new models for Internet services.

The battle of the LAN was won by Ethernet, the battle of the Edge is being won by Ethernet, so how far away is the battle for the core of the network? In fact it has already begun and for those whose network core router cheese is based on an old-style computer mainframe approach, it may move faster than you think. What will it do to networking equipment manufacturers? And, what will it do to network service providers?