Router Jockeys Beware - The System is Now Software

It's been a while, but I general try to limit my posts to larger issues and trends, and this one is as disruptive as the threat to truck drivers by self-driving Tesla tractor-trailers.

Back in late 2009, I wrote that the Internet is rapidly becoming a viable mechanism for Enterprise communication needs.  Eight years later, the growth of the Internet and Internet-based services have been more explosive than I every imagined.  The introduction of the Apple iPhone in 2007 led to dramatic changes in the entire Internet landscape, putting what are now billions of connected devices all chattering on the Internet for information services of tremendous variety.

These uses are of course, not limited to entertainment and the occasional check of stock prices, but whether at home or mobile, billions of dollars of commerce are directly related to the the ability of the Internet to provide both high-capacity and highly-reliable services.  Of course, the same companies that rely on the Internet for their customers, do not rely on the Internet for their internal mission communications.  Microsoft, Amazon, Google, and other have their own nationwide and international fiber networks that form that backbone of their businesses.

So for most enterprises, what are the options on the on the continuum of the Internet to a private fiber networks.  Until recently, an enterprise could build their network over the Internet (using IPSec tunnels, etc.) or over managed bandwidth services distinct from the Internet (e.g., common called MPLS/VPN services).  Of course, there is the combined case, where both can be combined.  For example, using Internet services (e.g., 4G wireless) as a backup in case the MPLS/VPN service is disrupted.

In both of these cases, a common denominator is the need to "build" the network out of network transport pieces provided by service providers.  The build has led one of the major costs and limits of recent network technology.  In general, we buy routers and then we need to buy the Router Jockey talent needed to create the configurations that make it all work.  Not only are the Router Jockeys needed, but a set of network management software will also be needed to configuration manage, monitor, and trend the resulting network.  Unfortunately, in many cases the Router Jockeys and Network Management Engineers are hardly on the same page.  Engineers like to tinker, and with the hundreds of configuration options on routers - with solutions driven by "feel" - network services may suffer.  Just try to figure out how to configure a network that uses a combination of MPLS/VPN, Internet, and 4G-Internet services to provide a robust network in the event of network failures.  After the Router Jockey talks for 30 minutes about Equal-Cost-Routing and MPLS labeling, go for a drink and know that there is a better way (don't even start with how to set-up all the IPSec tunnels you are going to need).

Software Defined Networks (SDN) was the better way that was supposed to come to the rescue.  An approach that  moves towards central control and management of network elements, should have been the answer.  But, again due to the complexity and lack of standardization, few enterprises have been able to take advantage of SDN.  

So, how about a new approach one that looks at an enterprise network as a complete system.  A solution in a box that understands end-to-end security, an ensemble of network devices that actually make a network, and does not care about the type of transport used to make the network.  Add Web-based control and RESTful APIs to integrate into an enterprises management system and you have the emerging world of Software Defined Wide Area Networking (SD-WAN).

SD-WAN has all the goodies built in to make network configuration faster, moving away from the Router Jockey to just another IT person configuring a IT system.  However, how does this address the issue of Internet services for Mission critical applications?

Tied directly into the SD-WAN approach is the addition of Packet Forward Error Correction (PFEC) technology and link bonding.  Unable to be performed by a standard router that only knows how to forward packets, PFEC uses very similar approach to the FEC found on optical communications systems.  PFEC sequences packets and adds additional overhead that enables in many cases the reconstruction packets dropped by the transport network (e.g., Internet) by the receiving SD-WAN device.  These techniques have existed for some time in WAN acceleration devices - however sitting on top of a standard network routers.  Finally, SD-WAN devices have combined PFEC and transport link bonding to use multiple transport services (e.g., MPLS/VPN, Internet, Satellite, 4G-based Internet) to provide amazingly robust end-to-end services.

Configured by a Web-page based on user needs and mission priorities, SD-WAN is going to continue the trend on what I wrote about in my early post driving Enterprises towards Internet-based transport and start a new trend - Router Jockeys are going to have to find a new link of work.

Good Grief, Isn’t Anyone Responsible Here?

We have all seen the news of the massive theft of information from the Office of Personnel Management (OPM).  In a nutshell, with extremely high probability just about anyone that does work for the government (or from one estimate over 21 million people), which includes yours truly, had very personal information stolen.  In my case, this could mean that the last 35 years of my life, everywhere I lived, everywhere I worked, the names and contact information of my close relatives and closest friends, and virtually everywhere I traveled outside of the United States of America is in the hands of what is speculated to be the Chinese government.  In some cases, of course other than myself, the information will include self-disclosed arrest information, drug and alcohol abuse disclosures, and whether bankruptcy was declared.  Good Grief! And, what do we get from those in charge of OPM? Well to my mind it is exactly what the Peanuts characters hear when the adults talk: "waa waa waa.".  Translated for your benefit: The (now former) Director of OPM says she does not believe "anyone is personally responsible".  It is just this lack of personal responsibility as well as other Cyber security failures that needs to motivate us to a new approach – one that recognizes that every business relationship today has its implementation foundation built on Information Technology (IT).

Although the director of OPM has now resigned, there is still no real accountability or responsibility.  Dozens of OPM government employees and contractors knew the state of their system and lack of protection.  The problem is that these people know, with certainty, that there is no accountability and there is no responsibility.  The Chief Information Office (CIO) has not resigned, and that person is directly accountable to Congress to represent that their systems are FISMA compliant.  Here is a link to the OPM Office of the Inspector General Report for 2014.  Just read the summary page under "What Did We Find?" and you will be appalled.  There is some glimmer of hope on the trail to real responsibility.  As reported in the Wall Street Journal CIO Report by Kim S. Nash, the OPM CIO better get some lawyers as she is being sued.  The legal threshold is high, but this is at least a step away from zero responsibility.  Of course, let’s say that she loses in court, what exactly is going to be the remedy for the people impacted (she most likely has no money, even if that is a potential judgement)?  Would this actually change the environment to get some real focus on Cyber security?

Of course it is easy to be a Monday Morning Quarterback and to Beat A Dead Horse, so let's move to a more constructive set of observations and advice.  The old adage is that you "get what you pay for".  In the business world it transforms into "you get what you measure", and I will contend that in the Cyber Security world "you get what someone is liable for".  In fact, the government gets precisely what is measures, preferring to award Lowest Cost Technically Acceptable (LCTA) contracts where in general there is no significant liability for Cyber failure and more importantly no emphasis on actually grading the contractors during source selection against Cyber security performance.  In the OPM case, the government hired a contractor to perform background checks and submit data.  This contractor’s system became entangled in the government’s system.  Because there was little emphasis on the Cyber security posture of the contractor as part of the performance of the contract, the contractor’s system was apparently not well managed or secured, and when the attackers found a hole, it ran all the way into the government and enabled the theft of massive amounts of data.  Most likely, the system was built by a contractor many years ago (of course this is speculation) and is still in place because budgets and priorities are always about maintaining status quo and "working on" new solutions.  In general, the government finds it significantly difficult to "abandon" outdated or obsolete systems, where industry does - it invests in the new modern methods and either sells of or disposes of the old.

Let me explain.  In the business environment companies have to directly address the risk of doing business.  This is represented by insurance for fire and theft, as well as in many cases for other business related issues such as product liability.  In the commercial world, poor business practices translate into higher business losses (for example product related liabilities) and an increase in costs due to rising insurance rates as well as potentially large expenses due to punitive damages imposed by a court decision.  In addition, business executives are directly accountable legally (e.g., Sarbanes-Oxley, HIPPA, etc.) and from their Boards and Stockholders - that is they lose their jobs and even can go to prison.  Business leaders are also responsible for the entirety of their business - accounting, hiring, delivery, liability, and profit to shareholders and owners.  Hence, they know how it all works together and they make decisions with the overall goal of sustaining the business and enabling growth as primary focus elements.

In the current government environment, and almost surely at OPM, none of the normal commercial business pressures are at play, especially in light of comments that "no one is personally responsible". The problem is that when the government takes on the responsibility directly, in general, there is virtually no administrative or legal repercussions for a massive failure.  Assuming good faith of effort, organizations such as OPM grow their government supervised internal Cyber Security operation setting-up processes, buying tools, and then trying to keep-up with the quickly morphing Cyber threat landscape.  Miss one step in this activity, and we get a massive Cyber failure.  Cyber security technology is not the culprit here - it is the lack of understanding by leadership as to how to apply Cyber security as an enterprise core competency where agency heads are not dazzled by the latest buzz words, but see the enterprise as a single architecture that includes its partners.

So, what can be done? A good friend of mine loves to quote Peter Drucker: "There is nothing so useless as doing efficiently that which should not be done at all".  In this case, doing more efficiently the internal Cyber efforts of a government organization nearly a waste, and the reason is simple.  Adding processes, tools, and oversight does not make anyone actually truly responsible or liable in the legal sense of the word.  What needs to be done is a complete shift of activity to an approach that selects providers that offer a warranty or service level agreement for not only the performance of the direct work (e.g., performing background checks) but also for all necessary associated IT components.  This is not just a selection by reputation, but a selection that is based on the company's willingness to "put their money where their mouth is".

The vision is that an organization like OPM will select a responsible party with a track record of performance that demonstrates that they can do the job and stand by their work when there is a Cyber event.  Sign them up for real metrics, for example on the time between discovery and reporting and for the number of days between major Cyber events.  Don't take their word for it, hire an independent auditor, review the Cyber performance every month (or week) and hold them to their agreements and hit them with penalties and even legal action for failure.  However, liability just is one element.  We must not perpetuate the "security is a separate function", so we need to do more.  We have to get away from escape clauses that boil down to the contractors "just doing what the government wants them to”, and where the government supervises or even performs the Assessment and Authorization (A&A) process for the systems.  Without these additional changes, the liability melts away into the political morass and standard government CYA.

The solution to our problem is that we need the system provider, and their subcontractors, to provide the "warranty" just as they do today for their financial systems.  Just as a company or the government may hire an accounting or financial firm today, they need to hire their Cyber firm - both need to be accepted and certified.  Then the government needs to focus on monitoring and spot checks, and not interfere in the contractor’s activities because when they do, the liability goes back to the unaccountable.

Of supreme importance, this needs to expand to include when the government contracts for virtually any service.  In OPM’s case the apparent root-cause system that enable the breach was associated with a contracted personnel background investigations company.  The fact that the performance of the contract came with an IT system that electronically interacted with the government's system is the point.  It does not matter what service or product you buy, you are buying into that company’s Cyber posture and how they manage their IT and how it interacts into agency’s IT.  This is what the CIO needs to understand and address.

You might ask whether anyone would take on such an activity, but it happens every day.  Certified Public Accountants and Professional Engineers have to sign their work.  Maybe it will scare away some of the "Johnny come lately" Cyber "expert" companies that give advice but take no responsibility for the actual result.  Companies that have their act together, that genuinely understand the risks and technology will rise to the occasion.

Using this method, there will never be a time when someone is not responsible.  True risk equals good reward for Cyber companies and other providers that actually stand up for their work combined with IT security results.  Maybe “doing efficiently” our current approach should “not be done at all” – after all it does not seem to be working.