Buckle-up, all infrastructure is software and your enterprise needs enterprise orchestration

The capabilities landscape for equipment manufacturers, service providers, and enterprises is rapidly changing, and within several years it will be fundamentally transformed from today.

There are several basic areas:

1. The creation of robust commercial Cloud services with a rich set of services all presented for allocation and configuration to the enterprise via a set of standard Application Programming Interfaces (APIs).
2. The emergence of the Software Defined Networking (SDN), offering the potential of flexible network services again presented to the enterprise for allocation and configuration as a set of APIs.
3. The transition of traditionally physical network-related devices to application that can be configured onto essentially standard computer servers, called Network Function Virtualization (NFV).

The battle that is the force driving these changes are between what I call the “new traditional” service providers and the “legacy” service providers.  Companies like Amazon and Google eschewed traditional wisdom of hardware providers and the paradigm of legacy service providers.  Driven by their application development and low-cost consumer mindset, their general approach is to strip-down to the necessary hardware and software functionality.  Bloated hardware and software with features and functionality not needed is removed.  The over 30 years of the evolution of Internet standards that defines to the control of network devices, embedded into expensive routers and switches, is discarded in part or whole for so called “white box” hardware and Open Source software as the basis for their control.

Legacy network service providers grew-up with the Internet, driving its standards within the common framework of a set of “autonomous systems” configured by the service provider with a set of defined end-user services.  Scant thought was given to providing end-users (in this case the enterprise customer) any meaningful end-to-end control of services, and almost without exception nothing that looks like a web-service RESTful API.  This is in stark contrast to the rich information and control APIs expected and provided by today’s commercial Clouds.

The figure below represents the recent past and much of the present.  Blue represents the legacy infrastructure approach.  Focusing on the network space, the enterprise has to contend with complicated device configurations and essentially static service configurations from their network services provider.  There is little if any coordination between the network and the applications development and operations environment other than at best service tickets and at worse verbal (and undocumented) direct staff-to-staff communications.

The expectations of enterprise Information Technology organizations will also drive the trend to a more software defined environment, as the use of Cloud services and it associated reporting and control will become the expectation, not the exception.  In fact, it is likely that more comprehensive “enterprise orchestration” systems will be developed that will cover all services, from internal application development lifecycles (i.e., Development and Operations), to control and management of end-to-end enterprise services delivery.

This leads to the view in the figure, below.  Red and green represent the new infrastructure trends and blue represents the legacy environment.  The significant change is that nearly all of the infrastructure is now software based, from SDN controlling and reporting of end-to-end network (including to and from Cloud resources), to the direct control of virtual network devices whether in the Cloud, at an enterprise location, mobile, or one of those Internet of Things devices using NFV.

When every resource or service is controlled by what appears to be a web-service and the same mechanism is used to obtain performance, usage, and other relevant from across the different traditional service domains (compute, storage, network, security, etc.) then everything looks like software.  Once this happens, one has to completely rethink an enterprise’s IT operation, as the same types of activity that is done to develop applications is now the fundamental discipline for orchestrating the enterprise, whether it is resource management, application development or rollout, or cyber security.

Buckle-up, time to become an enterprise orchestration programmer.

The Internet of Things and Active Digital Debris

We are starting to see an exponential increase in the amount of “Digital Debris” left behind from our romps through the digital world.  This debris contains not only personal information but also now represents active systems left on in the digital cloud and Internet of Things (IoT) wilderness.  To break down how we got here, we can structure this into several different epochs:

1. From the Digital Dawn to Shared Hosting Services
2. From Shared Hosting Services to the Cloud and the Dawn of the Internet of Things
3. To the Era of the Internet of Things (IoT)

Before we characterize each of these and the impact of the emerging IoT epoch is important to differentiate between two different types of debris:

1. Passive Digital Debris - This is characterized by both the offline and online digital data that we leave behind
2. Active Digital Debris - These are the active systems, that may have personal data, but more importantly are active in the control of something physical

Passive Digital Debris. In the first epoch, for those that can remember, information was stored on punched cards that were direct descendants from Herman Hollerith’s first machines produced for the U.S. Census Bureau in the 1890’s.  These 80 column paper records then moved to digital tape and then rapidly onto hard-disk drives.  Today, much of today’s consumer passive digital debris is embodied in the hard-drive of our desktops and laptops and the solid-state storage of our smartphones and pads (and of course, in landfills along with billions of biodegrading punch cards).

Since a person or organization should know the devices it has, it is relatively easy to clean-up this passive digital debris.  For devices that have hard-drives (spinning or solid-state) one can simply remove the drives and store them safely, or physically destroy them.  However, devices such as pads have embedded and very difficult to remove flash drives.  These are generally “wiped” before the devices are discarded, given away, or sold.  Unfortunately, this is easier said than done as is it is relatively complex to actually get rid of the information stored on these devices without some care.

So far, we have explored data that just lays around in devices.  It can not be accessed until attached to system connected to a network.  However, towards the end of the first epoch we see the emergence of dedicated and shared hosting services.  These provided the ability for people to create Websites and provide data services.  With simply a credit card an account could be set-up, website developed, and data uploaded.  Over time, how many thousands of these sites exist essentially unknown by their original owner, with the data littering the Web.

In the second epoch, with hosting and Cloud computing services, the situation now gets much worse.  With easy uploads of data to Cloud storage, whether through a managed service such as iCloud or more raw directly to a Cloud service (e.g., AWS S3 & Glacier or Google Cloud Storage), this data is being uploaded at dozens of Terabytes (probably much more, but I don’t want to sound too hyperbolic) a day.  With people canceling service, forgetting about the service they bought, or passing away, this data will stick around for month, years, and perhaps for whatever is humanity’s ultimate destiny.

Of prime importance is that this passive digital debris, although it may be accessible via a network does not directly interact with the physical world.  

Active Digital Debris.  Now, in the IoT epoch, IoT devices and systems create something new: Active Digital Debris.  Active Digital Debris are ensembles of those devices and their support ecosystems that become part of the long-lived infrastructure of a structure (e.g., home, business, car, etc).  For example, take the case where there IoT thermostats, refrigerators, lighting systems, an irrigation system, a security system, and several generations of digital cameras.  The original user that installed and configured the system understands (or thinks they understand) its use.  What happens when the house is sold?  What happens if the owner is no longer available?  What happens if the owner does not remember how the systems are configured or their passwords?  In fact, without an “IoT House Inspection ” how would a new homeowner even know what is lurking in the light bulb next to her bed?

So, there are significant questions on how do IoT systems transfer from owner to owner.  What are the responsibilities of a user to clean-up their Active Digital Debris?  Without exaggeration, within a few years there will be tens or hundreds of millions active devices within the homes, cars, and businesses that are essentially running against their last set of configurations, and unknown to the people they surround.  These devices may be the next trend in Cyber crime enabling illegal surveillance of home and bringing a new dimension to stalking.  In fact, there appears to be, what maybe is the first case of IoT-based revenge, where a spurned husband used an Internet connected thermostat to wage home temperature retribution against his apparently cheating wife (see, IoT Revenge).

Finally, what is the Active Digital Debris future?  It all depends on the emerging IoT ecosystems which is going to have to at least include mechanisms for some sort of consolidated inventory control and identity management approach.  Hmm, aren't these some of the holy grail of Information Technology? How about an IoT Pest Extermination Service?