Sunday, April 12, 2020

Buying Enterprise Network Services 2 - Understanding the Work at Hand


This blog is a continuation of my previous post Some Considerations for Buying Network Services. In this entry I am going to provide some thought material to help in understanding the characteristics of the different kinds of work an organization needs accomplished and some associated ideas on how to contract for the work.

Before we get started, one of the problems that I observe is that many organizations seem to deal with "absolutes".  They define a type of activity, for example "Networking", and then decide how to contract for the effort, for example "outsourcing".  Depending on the culture of the organization, this may lead to discussions that contain statements like "well, if  we outsource everything then we won't be able to tell the contractor anything and we get whatever they want to provide the service".  On the other extreme are statements like, "if we continue to use level-of-effort contracts with the government telling people what to do we won't get anywhere as no-one is actually responsible for making things better and we have no idea what all these people are actually doing".

Clearly, the extremes don't work.  So, what actually makes sense?  It can be pretty complicated, but one way to start is to understand what actually needs to get done.  From my experience there are three basic types of work:

  1. Services Outsourcing.  All IT organizations do this to an extent, but the question is really the need to understand the right level of what is a Service for an organization.  For networking, the lowest level is buying point-to-point telecommunication capacity and the highest level may be complex service types and levels delivered to the organization's locations with special services related to access to the Internet and Cloud services.  Importantly, Outsourcing contracts need to include specifications for the services to be delivered as well as the contract's interaction with an organization's other related activities.  Examples include, Enterprise IT Operations and Cyber Operations centers.
  2. Outcome-based Tasking.  This is the type of work that is designed to develop a capability based on a specification.  An example of this could be installation of a new local area network in a building to a set of performance and reliability specifications.
  3. Level of Effort Tasking.  This is the type of activity that maximizes flexibility.  An example is the rapid implementation of a new capability where there is very limited time,  only general requirements, and near real-time changes may be necessary to meet the actual mission need.

Some of the main characteristics of these different types of work of work are:

  • Specificity of the work.  This means that the definition of the services or work to be accomplished is much greater for Commodity Services Outsourcing than it is for Level of Effort activity.
  • Duration of the work.  Services Outsourcing are generally activities that span five to 10 years.  A Level of Effort activity might only last weeks.
  • Changes to the work.  Changes to Services Outsourcing generally requires the definition of a new service, negotiations with the contractor, and contract modifications.  All of this could take months.  Level of Effort activities can change on a near instantaneous basis.

A word on "absolutes" associated with Service Outsourcing.   Although, not as flexible as the other methods, it does not mean that a contractor can do whatever they want as long as they meet the contracted service levels.  Contract service levels is just one part of the what must be specified.  All the required parameters that affect the cost of the service must be included, as well as the verifications mechanisms required for proper oversight of the contract.

What does this all mean?  It means that the hard part of buying Network Services is to actually understand the services needed and all their related characteristics to address the diverse needs of the organization.  These are primary considerations in the types, requirements, and services levels that are part of the overall acquisition architecture, strategy, and roadmap.  More than a  technical issue,  it must also have what is in general more important, and many times neglected, a set of organizational structure and business decisions.

More on organizational and business elements in the next installment (hint: it is an architecture as well).

Monday, April 6, 2020

The new reality of home Internet usage




I occurred to me that I have not checked my home Internet usage in quite some time. In my last post on the topic of Internet home bandwidth usage in https://kaplowtech.blogspot.com/2011/09/more-usage-same-cost-boom.html, it appeared that my bandwidth usage was heading towards the then Comcast cap of 250 Gbytes per month.   My calculation was accurate as was Comcast's change of policy to remove the "cap" before the majority of their customers would have bombarded them with hate and discontent.

So, today I took a look at my home's Internet usage and remarkably the increase was less than 100% over period of the third quarter of 2011 to the first quarter of 2020.  We have Netflix, Prime, and do a fair amount of YouTube streaming, so I would think that our usage pattern is typical.  Of course, starting in September one of my Sons was off to college, so I am pretty sure usage was a bit higher in August 2019  before he moved to his dormitory .


Of course, now he is back and our family of four confined (mostly) to our home is banging on the Internet pretty hard.  We are clearly streaming up a storm and Teams'ing, Zoom'ing, and Webex'ing much of our work that our use increased by over 200% in one month.

So far, the performance of these services has been good.  It appears that the Internet is much more resilient than some people would have you believe or to give credit to the companies and their staff that build and operate the hundreds of networks that make it possible.  Of course, there was this BGP hijack, and a proliferation of cyber attacks, but that is another story…

Finally, I am not paying a proportionately higher Internet service bill than nearly 10 years ago.  What the future in pricing holds I do know know.   Perhaps new  services including wireless and new satellite systems may enable enough competition to keep prices down. 



Sunday, March 29, 2020

Some Considerations For Buying Enterprise Network Services


Under the current circumstances, and with acquisitions activities moving to the right, there is a bit of time to actually do a bit of thinking about how to meet enterprise network needs.  Maybe  some of our government organizations will take this additional time to consider how they could structure to improve their network acquisition and services performance.

If you get anything out of this post, I want you to ask yourself and your organization two simple questions:
  • "Are we buying our enterprise network service the best way possible?"
  • "Is my organization structured for effective enterprise network services acquisitions and operations?"

These simple but direct questions should lead you down a journey that will probably end with the conclusion that, no, you are not buying our network services the best way, and it's time for a change.

So, what I am going on about here?  Aren't there telecommunication leaders in the enterprise networking space that solve networking services for large enterprises?  Don't they provide the capabilities and options are critical to success?  Well, the real answer is not really a yes or a no.  This is because the answer is not just about them, it is also the business model that your organization selects to specify, acquire, and operate services.

First, here is a quick review of the major trends in telecommunications solutions for enterprise networks.

As I wrote previously, as far back as 2009, the expanding use of Internet-based services to meet Enterprise networking requirements continues.  The traditional dependency on a single service provider's MPLS/VPN network exclusively for bandwidth and to ensure quality of service is rapidly changing to a blend of services with Broadband Internet (i.e., the Cable company) and wireless (i.e., 4G and now 5G).  There are several factors driving this change: 
  • The cost of Broadband Internet services have a capacity per price ratio that depending on location ranges from two to as much as 100 times better than MPLS/VPN services (that is more bang for the network buck).
  • The quality of these services has improved dramatically (as proven today by millions of people working at home due to COVID-19 with the use of Internet access-based Software as a Service and high-quality video teleconferencing services).
  • The cost effective use of multiple services to improve effective network availability at a site due to a service failure (in most cases the share risk of failure between, for example, 4G/5G and Cable provided Internet is small).
  • SD-WAN technology is enabling policy-based use of  these services to improve the network performance experience of users and their applications.

Of course you say, my enterprise network service provider can do all this for me.  As a buying organization, we can pick a blend of my selected services provider's MPLS/VPN service as well as wireless and even Broadband Internet and I am good to do.  In addition, my selected provider offers SD-WAN service.   My organization is all set.

Well, you are, sort of.  As I have found in many areas of Information Technology delivery, it is more about the business model than technology.  If you are large commercial or government enterprise, these are some of the business areas and questions you should consider:
  • Are you a "one-and-done" organization?  That is do you perform a competitive acquisition activity once, set pricing, and then ride the solution for five to 10 years?
  • Are you an "outsourcing" organization?  That is, except for service requests, status, and trouble management you let your selected service provider to it all.
  • Are you an "engineering" organization?  That is, you have network engineers that want to design and engineer the network?

Each of these business processes lead to a different set of principles for buying network services.   Here are some to consider:
  • Do you want to avoid the complexity and effort required to perform a transition from the periodic "one and done" network service provider?
  • Are you concerned maintaining cost competitiveness during a "one and done" contract?  Global network services pricing, including U.S. domestic services, decreases every year.
  • Are you concerned that your organization's engineering staff can not keep up with a combination of technology and working with multiple vendors to pull a solution together and as important sustain the system?
  • Did you develop an operations architecture on how to integrate your total Information Technology (IT)environment?  For that matter do you have an IT architecture?
  • And most important, is the organization getting the network performance and availability needed to get the mission done with clear incentives for all parties to ensure continued performance.


The place to start is to create a set of objective outcomes for the organization as part of building a strategy:
  • Stop the need doing network acquisitions every several years (when you actually get around to putting together the acquisition documents, make an award, and perform a potentially costly and disruptive provider transition).
  • Enable the ability to continually shop for price and take advantage of reducing costs without complete redesign of the network.
  • Build an engineering staff that is fully integrated into a comprehensive business process that both leverages and builds their skills, and addresses the need to keep the organization's technology current.
  • Build a comprehensive network (and really IT) architecture that leverages a full range of network implementation and operations options tailored the various network services required.
  • Build and provide network services that meet and continue the organization's evolving needs.

This is a bunch to consider, and in future articles, I will focus on potential government organization structures and approaches to meet the above outcomes.

Sunday, February 10, 2019

Google Fiber Fail





I wrote way back in 2012 about Google Fiber.  Although that post focused on the marketing hype associated with Google providing 1Gbps Internet service, it did hint that Google was taking on physical infrastructure issues that could have major impacts to service.

The new installation techniques employed by Google and not tested by decades of real-world installations have come back to haunt Google and more importantly their complete customer base in Louisville, Kentucky.   Google attempted to take a shortcut to the installation of fiber with a technique that directly exposes fiber to the harsh realities of the environment.  This new method uses shallow slit trenching of fiber directly into the asphalt street pavement  and then it is covered with an epoxy - what could possibly go wrong?



Unlike concrete (which has its own issues),
Image result for google fiber louisville
asphalt is not solid or stable.  Asphalt moves and cracks. Years of layers of pavement generate layers that capture water, freeze and become potholes that are the bane of car tires everywhere.  The situation becomes worse depending on the combination of the type of ground under the pavement, weather, and very importantly they types and frequency of truck traffic.  What was the thought process that is approach was going to last a year, let alone the 10 to 20 year set-and-forget typically required to meet fiber or cable plant cost effectiveness?  You can actually see cracks and repairs in the pavement in the picture of Google installing fiber in Louisville.   Again, what were they thinking?

Although the Cable companies (the MSOs) use shortcut techniques of direct-bury of coaxial cable in the ground, this approach has cable systems that have the benefit of years of improvements and well known maintenance needs.  In general, although segments may fail, there are generally no regionally-wide systematic plant failures that require the complete re-installation of cable or fiber.

Google may have a fail-fast (or relatively fast if you are tracking Google's set of messaging applications) approach based on "Internet Time', but in this case, not only did they fail, there is no recovery enabled by the upgrade or download of an Android or iPhone app.  They created a false narrative that there was a shortcut to conventional installation approaches without performing the long-term testing that is the generally hallmark of stable and reliable telecommunications systems.

Finally, unlike other companies that plan for long-term commitments to their customers, Google Fiber is apparently leaving virtually all their Louisville customers forever.  Maybe Google will finally figure-out that although they can fix or abandon applications without significant damage to their main advertising-based revenue, failing at the physical layer to a customer is something that the customer will not soon forget.

Sunday, August 26, 2018

The Changing Landscape of Global Network Services



The landscape for World-wide network connectivity has been dominated by traditional carriers. Names like AT&T, Verizon, and BT. Although some of of the names may be relatively new (e.g., TATA, Orange) based on mergers, acquisitions, and business model expansions (e.g., moving from a regional to global model), the approach to providing services has generally followed a similar pattern. These are the traditional “telecoms”.

Driven initially by voice communications, the World was slowly, and then more rapidly connected by a series of every more capable Submarine Cable systems. The next phase was driven by multinational companies’ (including so called Enterprise customers) demand for data transport to meet their corporate needs and connect to their supply chain. With more demand, carriers responded by expanding their Points of Presence (PoPs) to new cities and creating consortia to build new Cable systems to increase network capacity and diversity.

The penultimate stage (at last from the perspective of where we are today), is the incredible impact of the Internet. With corporate data centers holding the family jewels located at private and commercial colocation facilities around the World, Internet traffic increased at unprecedented rates of growth driving facilities investment for colocation and hosting space as well as new consortium cable systems (as well as system upgrades based on advancements such as coherent optical technology). These networks, were (and are) provided by the appropriately named “Internet Service Providers” or ISPs.

The underlying business of the typical ISPs can been roughly seen by putting them into two groups. The first are relative “pure play” ISPs. These are companies that derive the majority of their revenue based on terrestrial communications, and therefore global expansion is part of their revenue expansion plans. The second group has more complicated revenue models, where the vast percentage of their revenue is based on their mobile wireless services in their home country (or region of countries). The reason this is important is the emphasis that these companies place on different aspects of their business.

The first group must expand their network services to attract commercial customers to their network. They have to show value and provide the customer expected high-touch support. The second group is at a crossroads. A large multi-country network may provide several millions of dollars a year in revenue, however it comes with associated high-cost to provide service. On the other hand, generating another 50,000 wireless customers (or wireless Internet of Things - IoT devices) will have the same effect, without the high-touch customer required, and it provides additional revenue for wireless services evolution (e.g., 5G upgrades).

While the scenario between ISPs plays out, a new set of companies with unprecedented growth is changing the landscape - these are the Cloud Service Providers (CSPs) - Amazon, Google, Microsoft, etc. It is now time for large global network consumers to evaluate new options for enterprise network services.

The typical customer understanding and use of CSPs is for their processing, storage, value added services, and Internet access. Thousands of companies have either moved part or all of their processing and storage (a.k.a., hosting) needs to CSPs. The major CSPs are seeing 40% revenue growth per quarter and revenue now runs billions of dollars a month. This vastly outstrips the growth of traditional corporate enterprise network growth and total revenue of the ISPs. This reality has created two effects:

  • CSPs have created their own network infrastructure, building nationwide networks, consuming huge amounts of existing, and now building their own, Submarine Cable systems (more on this later) - all of this to handle something on the order of 50-70% of all Internet traffic with extraordinary network diversity and robustness
  • World-wide and regional ISPs have responded to their Enterprise customer’s need to use CSP resources by extending their ISP networks, bringing Internet peering and MPLS/VPN services directly into CSP facilities
The result of this is that CSPs are becoming the most capable global network infrastructures by far. As they expand in scale (more capabilities at existing CSP locations) and scope (more CSP locations), this capability will only increase. Feeding this monster, that is getting customer access into place, will be an ever expanding set of capabilities from local, regional, and national wireline and wireless networks.

So, what does this mean to the large global company that needs robust and cost effective communications around the World? It means that there are new technical and business approach options that need to be considered.

On the technical side, these companies need to:
  • Map their geographic network requirements against the major CSP data centers (regions)
  • Understand the CSPs inter-region services and their cost structure
  • Understand the global and regional network providers that have a PoP in the CSP regions
  • Develop and approach to leverage the CSP’s virtual services to develop an Enterprise network backbone that can use the network services at the CSP’s location (including Internet access)
  • Develop an approach to securely leverage multiple local access providers by using Software Defined Wide Area Networking - SD-WAN. This includes regional MPLS/VPN providers (e.g., MPLS/VPN), 4G and emerging 5G wireless, Internet services, and satellite services and integrate into the developed Enterprise network backbone
  • Understand each Enterprise site’s service needs and cost trades:

  1. How much bandwidth?
  2. What service resiliency is required?
  3. How long a service interruption can be tolerated?
  4. Make the trade between expensive MPLS/VPN and cheaper Internet bandwidth (see Yikes, Internet for Enterprise Services)

  • Map out the regional networks that could support each site’s service needs
On the business side:
  • Understand the opportunity of leveraging the CSP for the Enterprise network backbone combined with the traditional Cloud services to meet the Enterprise’s needs
  • Understand the complexity of performing as your own multi-vendor integrator, buying network services from multiple carriers to provide Internet and MPLS/VPN services needed to connect to the CSP-based Enterprise backbone
In summary, this is a time of significant change all every level of the Information Technology “Stack”. From how to operate and build a network (using Software Defined Network - SDN orchestration and SD-WAN), virtualization of devices (using Network Function Virtualization - NFV), and network resources, to how to balance Enterprise on-premises hosting with CSP-based services.

Understanding these new capabilities and navigating the complexity to create the high-performance and cost-effective network services for an Enterprise to be globally competitive is the challenge.

Sunday, September 24, 2017

Stolen Data, Internet Giants, Social Media, and Artificial Intelligence: The Growing IoT Risk

You can scarcely read the news today and not find an article about an information breach of one sort or another.  The most visible one today is the Equifax breach.  Exposing the personal information of nearly 150 million customers, much of the concern and recommended actions are focused on protecting  people’s credit by placing a “credit freeze” on accounts, but should this be the only concern? For Equifax their business goal is apparently not just to gather the information needed to address the credit worthiness of a person or corporation, but to amass information that it could productize to sell to other companies, including ironically information used to address Cyber security breaches.

Are they the only company amassing unprecedented concentrations of personal data? Is the credit report of the average person really the biggest potential problem?  The answer to both is definitively no.

All of us are complicit in the willful offering of our personal information to the mega Internet companies, being enticed by free services and our own vanity.  It is precisely this information value that drives Internet advertising and sales support making companies worth hundreds of billions of dollars.  These services contain not only static information but also patterns-of-life of hundreds of millions of people.

What is the danger, why do we believe that these companies or really any company can protect its information from hacking exfiltration?  How do we understand the risk of Insider Threats of these great compilations of data?  Will we even know when breaches occur?  Do we know when it is for monetary gain or even more concerning for political or military power?

With the explosion of Artificial Intelligence (AI) systems, what is the training set needed before an AI understands a person’s life and puts together the details and information that today forms the basis of identity and trust?  And then use of this information for nefarious purposes.

So, we worry about the opening of a new unauthorized credit account, but do we worry about the modification of an account that enables the unauthorized opening of a door in a house, or turning on off lights or heating?  In a few years, virtually all cars will be connected, how many cars need to be “hijacked” before chaos ensues?

These are a lot of questions, but they are some of the critical questions that need to be considered as the Internet of Things becomes virtually everything and our current ability to securely operate these systems gets away from us.

I hope to see you at the IEEE CNS 2017 Industry Track where we will expand and hopefully see some answers.

Sunday, April 30, 2017

Router Jockeys Beware - The System is Now Software

It's been a while, but I general try to limit my posts to larger issues and trends, and this one is as disruptive as the threat to truck drivers by self-driving Tesla tractor-trailers.

Back in late 2009, I wrote that the Internet is rapidly becoming a viable mechanism for Enterprise communication needs.  Eight years later, the growth of the Internet and Internet-based services have been more explosive than I every imagined.  The introduction of the Apple iPhone in 2007 led to dramatic changes in the entire Internet landscape, putting what are now billions of connected devices all chattering on the Internet for information services of tremendous variety.

These uses are of course, not limited to entertainment and the occasional check of stock prices, but whether at home or mobile, billions of dollars of commerce are directly related to the the ability of the Internet to provide both high-capacity and highly-reliable services.  Of course, the same companies that rely on the Internet for their customers, do not rely on the Internet for their internal mission communications.  Microsoft, Amazon, Google, and other have their own nationwide and international fiber networks that form that backbone of their businesses.

So for most enterprises, what are the options on the on the continuum of the Internet to a private fiber networks.  Until recently, an enterprise could build their network over the Internet (using IPSec tunnels, etc.) or over managed bandwidth services distinct from the Internet (e.g., common called MPLS/VPN services).  Of course, there is the combined case, where both can be combined.  For example, using Internet services (e.g., 4G wireless) as a backup in case the MPLS/VPN service is disrupted.

In both of these cases, a common denominator is the need to "build" the network out of network transport pieces provided by service providers.  The build has led one of the major costs and limits of recent network technology.  In general, we buy routers and then we need to buy the Router Jockey talent needed to create the configurations that make it all work.  Not only are the Router Jockeys needed, but a set of network management software will also be needed to configuration manage, monitor, and trend the resulting network.  Unfortunately, in many cases the Router Jockeys and Network Management Engineers are hardly on the same page.  Engineers like to tinker, and with the hundreds of configuration options on routers - with solutions driven by "feel" - network services may suffer.  Just try to figure out how to configure a network that uses a combination of MPLS/VPN, Internet, and 4G-Internet services to provide a robust network in the event of network failures.  After the Router Jockey talks for 30 minutes about Equal-Cost-Routing and MPLS labeling, go for a drink and know that there is a better way (don't even start with how to set-up all the IPSec tunnels you are going to need).

Software Defined Networks (SDN) was the better way that was supposed to come to the rescue.  An approach that  moves towards central control and management of network elements, should have been the answer.  But, again due to the complexity and lack of standardization, few enterprises have been able to take advantage of SDN.  

So, how about a new approach one that looks at an enterprise network as a complete system.  A solution in a box that understands end-to-end security, an ensemble of network devices that actually make a network, and does not care about the type of transport used to make the network.  Add Web-based control and RESTful APIs to integrate into an enterprises management system and you have the emerging world of Software Defined Wide Area Networking (SD-WAN).

SD-WAN has all the goodies built in to make network configuration faster, moving away from the Router Jockey to just another IT person configuring a IT system.  However, how does this address the issue of Internet services for Mission critical applications?

Tied directly into the SD-WAN approach is the addition of Packet Forward Error Correction (PFEC) technology and link bonding.  Unable to be performed by a standard router that only knows how to forward packets, PFEC uses very similar approach to the FEC found on optical communications systems.  PFEC sequences packets and adds additional overhead that enables in many cases the reconstruction packets dropped by the transport network (e.g., Internet) by the receiving SD-WAN device.  These techniques have existed for some time in WAN acceleration devices - however sitting on top of a standard network routers.  Finally, SD-WAN devices have combined PFEC and transport link bonding to use multiple transport services (e.g., MPLS/VPN, Internet, Satellite, 4G-based Internet) to provide amazingly robust end-to-end services.

Configured by a Web-page based on user needs and mission priorities, SD-WAN is going to continue the trend on what I wrote about in my early post driving Enterprises towards Internet-based transport and start a new trend - Router Jockeys are going to have to find a new link of work.





Wednesday, July 15, 2015

Good Grief, Isn’t Anyone Responsible Here?

We have all seen the news of the massive theft of information from the Office of Personnel Management (OPM).  In a nutshell, with extremely high probability just about anyone that does work for the government (or from one estimate over 21 million people), which includes yours truly, had very personal information stolen.  In my case, this could mean that the last 35 years of my life, everywhere I lived, everywhere I worked, the names and contact information of my close relatives and closest friends, and virtually everywhere I traveled outside of the United States of America is in the hands of what is speculated to be the Chinese government.  In some cases, of course other than myself, the information will include self-disclosed arrest information, drug and alcohol abuse disclosures, and whether bankruptcy was declared.  Good Grief! And, what do we get from those in charge of OPM? Well to my mind it is exactly what the Peanuts characters hear when the adults talk: "waa waa waa.".  Translated for your benefit: The (now former) Director of OPM says she does not believe "anyone is personally responsible".  It is just this lack of personal responsibility as well as other Cyber security failures that needs to motivate us to a new approach – one that recognizes that every business relationship today has its implementation foundation built on Information Technology (IT).

Although the director of OPM has now resigned, there is still no real accountability or responsibility.  Dozens of OPM government employees and contractors knew the state of their system and lack of protection.  The problem is that these people know, with certainty, that there is no accountability and there is no responsibility.  The Chief Information Office (CIO) has not resigned, and that person is directly accountable to Congress to represent that their systems are FISMA compliant.  Here is a link to the OPM Office of the Inspector General Report for 2014.  Just read the summary page under "What Did We Find?" and you will be appalled.  There is some glimmer of hope on the trail to real responsibility.  As reported in the Wall Street Journal CIO Report by Kim S. Nash, the OPM CIO better get some lawyers as she is being sued.  The legal threshold is high, but this is at least a step away from zero responsibility.  Of course, let’s say that she loses in court, what exactly is going to be the remedy for the people impacted (she most likely has no money, even if that is a potential judgement)?  Would this actually change the environment to get some real focus on Cyber security?

Of course it is easy to be a Monday Morning Quarterback and to Beat A Dead Horse, so let's move to a more constructive set of observations and advice.  The old adage is that you "get what you pay for".  In the business world it transforms into "you get what you measure", and I will contend that in the Cyber Security world "you get what someone is liable for".  In fact, the government gets precisely what is measures, preferring to award Lowest Cost Technically Acceptable (LCTA) contracts where in general there is no significant liability for Cyber failure and more importantly no emphasis on actually grading the contractors during source selection against Cyber security performance.  In the OPM case, the government hired a contractor to perform background checks and submit data.  This contractor’s system became entangled in the government’s system.  Because there was little emphasis on the Cyber security posture of the contractor as part of the performance of the contract, the contractor’s system was apparently not well managed or secured, and when the attackers found a hole, it ran all the way into the government and enabled the theft of massive amounts of data.  Most likely, the system was built by a contractor many years ago (of course this is speculation) and is still in place because budgets and priorities are always about maintaining status quo and "working on" new solutions.  In general, the government finds it significantly difficult to "abandon" outdated or obsolete systems, where industry does - it invests in the new modern methods and either sells of or disposes of the old.

Let me explain.  In the business environment companies have to directly address the risk of doing business.  This is represented by insurance for fire and theft, as well as in many cases for other business related issues such as product liability.  In the commercial world, poor business practices translate into higher business losses (for example product related liabilities) and an increase in costs due to rising insurance rates as well as potentially large expenses due to punitive damages imposed by a court decision.  In addition, business executives are directly accountable legally (e.g., Sarbanes-Oxley, HIPPA, etc.) and from their Boards and Stockholders - that is they lose their jobs and even can go to prison.  Business leaders are also responsible for the entirety of their business - accounting, hiring, delivery, liability, and profit to shareholders and owners.  Hence, they know how it all works together and they make decisions with the overall goal of sustaining the business and enabling growth as primary focus elements.

In the current government environment, and almost surely at OPM, none of the normal commercial business pressures are at play, especially in light of comments that "no one is personally responsible". The problem is that when the government takes on the responsibility directly, in general, there is virtually no administrative or legal repercussions for a massive failure.  Assuming good faith of effort, organizations such as OPM grow their government supervised internal Cyber Security operation setting-up processes, buying tools, and then trying to keep-up with the quickly morphing Cyber threat landscape.  Miss one step in this activity, and we get a massive Cyber failure.  Cyber security technology is not the culprit here - it is the lack of understanding by leadership as to how to apply Cyber security as an enterprise core competency where agency heads are not dazzled by the latest buzz words, but see the enterprise as a single architecture that includes its partners.

So, what can be done? A good friend of mine loves to quote Peter Drucker: "There is nothing so useless as doing efficiently that which should not be done at all".  In this case, doing more efficiently the internal Cyber efforts of a government organization nearly a waste, and the reason is simple.  Adding processes, tools, and oversight does not make anyone actually truly responsible or liable in the legal sense of the word.  What needs to be done is a complete shift of activity to an approach that selects providers that offer a warranty or service level agreement for not only the performance of the direct work (e.g., performing background checks) but also for all necessary associated IT components.  This is not just a selection by reputation, but a selection that is based on the company's willingness to "put their money where their mouth is".

The vision is that an organization like OPM will select a responsible party with a track record of performance that demonstrates that they can do the job and stand by their work when there is a Cyber event.  Sign them up for real metrics, for example on the time between discovery and reporting and for the number of days between major Cyber events.  Don't take their word for it, hire an independent auditor, review the Cyber performance every month (or week) and hold them to their agreements and hit them with penalties and even legal action for failure.  However, liability just is one element.  We must not perpetuate the "security is a separate function", so we need to do more.  We have to get away from escape clauses that boil down to the contractors "just doing what the government wants them to”, and where the government supervises or even performs the Assessment and Authorization (A&A) process for the systems.  Without these additional changes, the liability melts away into the political morass and standard government CYA.

The solution to our problem is that we need the system provider, and their subcontractors, to provide the "warranty" just as they do today for their financial systems.  Just as a company or the government may hire an accounting or financial firm today, they need to hire their Cyber firm - both need to be accepted and certified.  Then the government needs to focus on monitoring and spot checks, and not interfere in the contractor’s activities because when they do, the liability goes back to the unaccountable.

Of supreme importance, this needs to expand to include when the government contracts for virtually any service.  In OPM’s case the apparent root-cause system that enable the breach was associated with a contracted personnel background investigations company.  The fact that the performance of the contract came with an IT system that electronically interacted with the government's system is the point.  It does not matter what service or product you buy, you are buying into that company’s Cyber posture and how they manage their IT and how it interacts into agency’s IT.  This is what the CIO needs to understand and address.

You might ask whether anyone would take on such an activity, but it happens every day.  Certified Public Accountants and Professional Engineers have to sign their work.  Maybe it will scare away some of the "Johnny come lately" Cyber "expert" companies that give advice but take no responsibility for the actual result.  Companies that have their act together, that genuinely understand the risks and technology will rise to the occasion.

Using this method, there will never be a time when someone is not responsible.  True risk equals good reward for Cyber companies and other providers that actually stand up for their work combined with IT security results.  Maybe “doing efficiently” our current approach should “not be done at all” – after all it does not seem to be working.

Monday, March 30, 2015

Buckle-up, all infrastructure is software and your enterprise needs enterprise orchestration

The capabilities landscape for equipment manufacturers, service providers, and enterprises is rapidly changing, and within several years it will be fundamentally transformed from today.

There are several basic areas:
  1. The creation of robust commercial Cloud services with a rich set of services all presented for allocation and configuration to the enterprise via a set of standard Application Programming Interfaces (APIs).
  2. The emergence of the Software Defined Networking (SDN), offering the potential of flexible network services again presented to the enterprise for allocation and configuration as a set of APIs.
  3. The transition of traditionally physical network-related devices to application that can be configured onto essentially standard computer servers, called Network Function Virtualization (NFV).

The battle that is the force driving these changes are between what I call the “new traditional” service providers and the “legacy” service providers.  Companies like Amazon and Google eschewed traditional wisdom of hardware providers and the paradigm of legacy service providers.  Driven by their application development and low-cost consumer mindset, their general approach is to strip-down to the necessary hardware and software functionality.  Bloated hardware and software with features and functionality not needed is removed.  The over 30 years of the evolution of Internet standards that defines to the control of network devices, embedded into expensive routers and switches, is discarded in part or whole for so called “white box” hardware and Open Source software as the basis for their control.

Legacy network service providers grew-up with the Internet, driving its standards within the common framework of a set of “autonomous systems” configured by the service provider with a set of defined end-user services.  Scant thought was given to providing end-users (in this case the enterprise customer) any meaningful end-to-end control of services, and almost without exception nothing that looks like a web-service RESTful API.  This is in stark contrast to the rich information and control APIs expected and provided by today’s commercial Clouds.

The figure below represents the recent past and much of the present.  Blue represents the legacy infrastructure approach.  Focusing on the network space, the enterprise has to contend with complicated device configurations and essentially static service configurations from their network services provider.  There is little if any coordination between the network and the applications development and operations environment other than at best service tickets and at worse verbal (and undocumented) direct staff-to-staff communications.
The expectations of enterprise Information Technology organizations will also drive the trend to a more software defined environment, as the use of Cloud services and it associated reporting and control will become the expectation, not the exception.  In fact, it is likely that more comprehensive “enterprise orchestration” systems will be developed that will cover all services, from internal application development lifecycles (i.e., Development and Operations), to control and management of end-to-end enterprise services delivery.

This leads to the view in the figure, below.  Red and green represent the new infrastructure trends and blue represents the legacy environment.  The significant change is that nearly all of the infrastructure is now software based, from SDN controlling and reporting of end-to-end network (including to and from Cloud resources), to the direct control of virtual network devices whether in the Cloud, at an enterprise location, mobile, or one of those Internet of Things devices using NFV.
When every resource or service is controlled by what appears to be a web-service and the same mechanism is used to obtain performance, usage, and other relevant from across the different traditional service domains (compute, storage, network, security, etc.) then everything looks like software.  Once this happens, one has to completely rethink an enterprise’s IT operation, as the same types of activity that is done to develop applications is now the fundamental discipline for orchestrating the enterprise, whether it is resource management, application development or rollout, or cyber security.

Buckle-up, time to become an enterprise orchestration programmer.

Wednesday, December 31, 2014

The Internet of Things and Active Digital Debris

We are starting to see an exponential increase in the amount of “Digital Debris” left behind from our romps through the digital world.  This debris contains not only personal information but also now represents active systems left on in the digital cloud and Internet of Things (IoT) wilderness.  To break down how we got here, we can structure this into several different epochs:


  1. From the Digital Dawn to Shared Hosting Services
  2. From Shared Hosting Services to the Cloud and the Dawn of the Internet of Things
  3. To the Era of the Internet of Things (IoT)


Before we characterize each of these and the impact of the emerging IoT epoch is important to differentiate between two different types of debris:


  1. Passive Digital Debris - This is characterized by both the offline and online digital data that we leave behind
  2. Active Digital Debris - These are the active systems, that may have personal data, but more importantly are active in the control of something physical


Passive Digital Debris. In the first epoch, for those that can remember, information was stored on punched cards that were direct descendants from Herman Hollerith’s first machines produced for the U.S. Census Bureau in the 1890’s.  These 80 column paper records then moved to digital tape and then rapidly onto hard-disk drives.  Today, much of today’s consumer passive digital debris is embodied in the hard-drive of our desktops and laptops and the solid-state storage of our smartphones and pads (and of course, in landfills along with billions of biodegrading punch cards).


Since a person or organization should know the devices it has, it is relatively easy to clean-up this passive digital debris.  For devices that have hard-drives (spinning or solid-state) one can simply remove the drives and store them safely, or physically destroy them.  However, devices such as pads have embedded and very difficult to remove flash drives.  These are generally “wiped” before the devices are discarded, given away, or sold.  Unfortunately, this is easier said than done as is it is relatively complex to actually get rid of the information stored on these devices without some care.


So far, we have explored data that just lays around in devices.  It can not be accessed until attached to system connected to a network.  However, towards the end of the first epoch we see the emergence of dedicated and shared hosting services.  These provided the ability for people to create Websites and provide data services.  With simply a credit card an account could be set-up, website developed, and data uploaded.  Over time, how many thousands of these sites exist essentially unknown by their original owner, with the data littering the Web.


In the second epoch, with hosting and Cloud computing services, the situation now gets much worse.  With easy uploads of data to Cloud storage, whether through a managed service such as iCloud or more raw directly to a Cloud service (e.g., AWS S3 & Glacier or Google Cloud Storage), this data is being uploaded at dozens of Terabytes (probably much more, but I don’t want to sound too hyperbolic) a day.  With people canceling service, forgetting about the service they bought, or passing away, this data will stick around for month, years, and perhaps for whatever is humanity’s ultimate destiny.


Of prime importance is that this passive digital debris, although it may be accessible via a network does not directly interact with the physical world.  


Active Digital Debris.  Now, in the IoT epoch, IoT devices and systems create something new: Active Digital Debris.  Active Digital Debris are ensembles of those devices and their support ecosystems that become part of the long-lived infrastructure of a structure (e.g., home, business, car, etc).  For example, take the case where there IoT thermostats, refrigerators, lighting systems, an irrigation system, a security system, and several generations of digital cameras.  The original user that installed and configured the system understands (or thinks they understand) its use.  What happens when the house is sold?  What happens if the owner is no longer available?  What happens if the owner does not remember how the systems are configured or their passwords?  In fact, without an “IoT House Inspection ” how would a new homeowner even know what is lurking in the light bulb next to her bed?


So, there are significant questions on how do IoT systems transfer from owner to owner.  What are the responsibilities of a user to clean-up their Active Digital Debris?  Without exaggeration, within a few years there will be tens or hundreds of millions active devices within the homes, cars, and businesses that are essentially running against their last set of configurations, and unknown to the people they surround.  These devices may be the next trend in Cyber crime enabling illegal surveillance of home and bringing a new dimension to stalking.  In fact, there appears to be, what maybe is the first case of IoT-based revenge, where a spurned husband used an Internet connected thermostat to wage home temperature retribution against his apparently cheating wife (see, IoT Revenge).


Finally, what is the Active Digital Debris future?  It all depends on the emerging IoT ecosystems which is going to have to at least include mechanisms for some sort of consolidated inventory control and identity management approach.  Hmm, aren't these some of the holy grail of Information Technology? How about an IoT Pest Extermination Service?

Sunday, July 20, 2014

Comments on Amazing Predictions from 30 Years Ago...

I have a decent collection of classic texts on Computer Science.  These include both books on the history of computers as well as some (well, maybe too many) textbooks from my academic career.  On a fairly regular basis, I take a book off the shelf to see what I can glean from previous wisdom.  In general, I am looking for those technical and business decisions that enabled the rise or fall of technologies and companies.  This time, I took “An Introduction to Operating Systems”, by Harvey M. Deitel, Revised Edition 1984, a book thirty years old.  A classic at the time, the book spends much of its time talking about operating system concepts and issues that most people, and even most newer Computer Science students, do not even know were at one time fundamental trades in the evolution of computer Operating Systems.  For example, how many people care or understand about allocation of memory in a multiprogramming environment with a fixed number of tasks and different memory partition as compared to variable memory partitions?


However, what is truly remarkable about the book is actually in the Introduction section “Future Trends”.  I am going to look at a selection and and make some modern-day comments as well as some comments on what we (or I) should have tracked for investment opportunities.


  1. Computer hardware will continue to decline in price, while processor speeds increase, storage capacities increase, and the physical size of processors and memories decrease.  
    • This trend was clearly obvious in 1984
    • Back in 1984 the Intel 80386 was just about to be released which at 16 MHz provided about 5 MIPS of 32-bit processing.  Today a 2+GHz single core processor provides around two Billion Operations Per Second
    • A 1GByte hard drive was around $50K.  Today, a 1 TByte disk is less than $100
    • What took a room for a single mainframe or minicomputer of the day is now a room with hundreds of servers and thousands of cores (and, thousands of servers and hundreds of thousands of cores)
    • See the next trend for investment comments
  2. The “scale of integration” will continue to increase with VLSI moving to ULSI over the next decade.
    • The 80386 had approximately 275,000 transistors
    • Today, modern multi-core processors and Graphics Processing Units have several billion transistors
    • Investment: The trend should have been to look at companies that were developing microprocessors that would sell in the millions in applications areas that were fantasies at the time.
  3. Massive parallelism will become more common.  It will become possible to execute parallel programs with great speed because of the very high degree of concurrency.
    • Today we have parallelism at many levels: multithreaded cores, multi-core sockets, multi-socket servers, GPUs with hundreds of processors, and machines with millions of cores.
    • Take a look at the TOP500 supercomputers in the world and the massive problems set sides and complexity enabled by massive parallelism.
    • Virtually every large commercial web service reflects a massively parallel execution of requests and data analysis.
    • Investment: Companies that were developing novel technologies to incorporate dozens or hundreds of processors for applications, such as high-performance graphics, that would find themselves into workstations and consumer devices.  Software companies that were looking to orchestrate huge amounts of data.  For software, the interesting trend, is the proliferation of Open Source-based products, which enabled great capability in working with large data, but potentially dilute the revenue potential across multiple companies providing very similar products.
  4. Computers and their operating systems will be designed to foster the operation of virtual machines.  Real machines will be hidden from the user.
    • IBM invented the Virtual Machine concept that was embodied in their VM/370 system.  It wallowed for some time as something that was invisible to the larger computer industry, but with the proliferation of Virtual Machine systems for several different product lines (e.g., Intel, SPARC, z-series, etc.) virtual machines are even common on laptops.
    • These systems are also enabled by the incorporation of specific capabilities into the processor set.
    • Investment: Clearly VMWare comes to mind.  Today, you would have a nearly 100% return on investment if bought on its opening day in 2007.
  5. The family concept of computers as first introduced in the IBM 360 series will endure.  Users will be able to run their applications on many different members of a family of computers, and these applications will see only virtual machines.
    • For better or for worse, applications written for older versions of Microsoft Windows will run on newer version of Windows.  The same can be said of Android applications, and Apple iOS and Desktop operating systems (with some major bumps due to 68000 to PowerPC to x86 changes).
    • More and more, the core services provided via the Web or in the corporate data center runs in Virtual Machines.
    • Investment: This is the realization of the emergence of “ecosystem” economics.  Its about the breadth and sustainment of developers by enabling long-term stable platforms for users.  Microsoft, Apple, Google, Intel, and ARM are good examples.  Those that have not maintained a good ecosystem have generally struggled (e.g, Blackberry)  
  6. As new generations of computers are introduced by the manufacturers, existing programs will run on the new equipment as is, or with nominal conversion effort.
    • This is an amazing fact that user code written for Windows machines or under Linux operating system move virtually untouched as computers incorporate updated processors and other system improvements (e.g., RAM, storage, graphics acceleration, etc.)
    • This is also true in the Apple iOS and Android worlds
    • Investment: This is directly tied to the emergency of the major ecosystems
  7. The cost of data communications will continue to decrease, and data transmission speeds will increase.
    • Although a not rocket science production (technology makes things better and cost less), the typical approximately 1.5Mbps point-to-point circuit in the continental United States was around $10,000 per month in 1984 and a phone call (the equivalent of around 30Kbps, if you have the technology) was well over 10 cents per minute.
    • Today, an Internet connection at home typically costs around $10 per Mbps (if you average upload and download speeds) and commercial rates for high-capacity links are less than one dollar a Mbps.  And, of course more than a simple private line, the Internet brings you a network around the world that touches on the order of a billion devices.
    • Investment: Well, there was that telecom bubble, but if you bought Cisco stock at an equivalent of $0.04 in 1990 your return today would be over 62,000%.
  8. Computers will be tied increasingly into networks, and work performed for a user may be done on a computer of which the user is unaware.  This will continue to emphasize the importance of the view of virtual machines.
    • Can someone say “Cloud”.  This is clearly a homerun prediction.  Both in the fact that I am using Google Apps to write this document that uses a set of servers that I have absolutely no idea of what or where they are.
    • Investment: This addresses both infrastructure capabilities and content.  Google, Amazon, Microsoft, Rackspace, and others.  Google return since its IPO is approximately 1200% (in 10 years).
  9. Personal computers will be omnipresent.  Utilization of the resource will mean less than its availability, reliability, flexibility, and “user friendliness”.
    • One of the prevailing problems before the 80s was the full utilization of expense centrally located mainframe-type computers.  The trend to multi-user minicomputers was well underway, and the personal computer revolution was getting started in earnest.  The Apple Mac was released in 1984 and “user friendliness” has been the key design element ever since.
    • Desktops, Laptops, Tablets, and Mobile devices are the norm.  The goal is usability, and the fact that you don’t keep the four ARM cores in your tablet busy all the time is virtually no concern to anyone.  The fact that they are available when needed to ensure the near instantaneous responsiveness of the user interface or an application to make the user happy, is what it is all about.
    • Investment: Clearly, if you follow the “user friendliness” as the primary factor, then all eyes should have been on Apple, the company that focused on the “Insanely Great” user experience.  From 1984 to today, this would have grossed a handy 2,300% return.