Big Data

AWS re:Invent Andy Jassy Keynote – IT Application Services

Amazon Web Services (AWS) CEO, Andy Jassy re:Invent keynote was a mixture of new and updated core IT compute functions and the announcement of several new IT services that can be used to create new customer experiences. In my previous post I reviewed the core IT compute function announcements here. I believe this year AWS's focus has matured from a pure IT infrastructure (compute, network, storage) service provider to now provide easily consumed application infrastructure services that will accelerate the development new classes of applications focused on improving customer experiences.

AWS is on pace to release 1,000 new services and functions in 2016. Andy noted that on average an AWS user has three new services or function available to them each day. The pace of AWS innovation is impressive. This year AWS has begun introducing more application services that will change how customers interact with businesses.


AppCycleOver the past few years we have seen an explosion in the amount of digital data that is generated, and stored by applications. Many of the same businesses have struggled using this data to improve efficiency, and improve customer experience. Big Data analytics has required specialized skill sets and complex new tools that are needed to analyze the new data. AWS introduced a set of new data analytics services to make it easier for most businesses to analyze large volumes and variety of data quickly. These services simplify modern data analytics by eliminating the need for complex tool setup, and operations.

In 2015, AWS launched a standard relational database service, Aurora. The Aurora service has become the fastest growing AWS service. Andy reported 14,000 databases have been migrated from commercial (SQLServer, Oracle) relational databases to Aurora. This year PostgreSQL compatibility was added in addition to MySQL support.

Unstructured data (files) is the fastest growing data type for enterprise IT. Most customer invoices, manufacturing manifests, and transaction logs are stored as unstructured file data. AWS customers typically store their unstructured data in the Simple Storage Service (S3). A new service, Athena enabling standard SQL queries to analyze data stored on S3. Traditionally, analyzing unstructured data required migration to a specialized file system such as HDFS, and the setup of complex, and specialized analytics software. With the Athena service no setup is required to analyze data stored on S3. This will enable businesses to easily analyze large structured and unstructured data with familiar tools (SQL). This will allow business to create new data based efficiency and customer experiences faster and more cost effectively.

Image Processing

As more customer experiences move online, it is critical to apply image analysis application capability. Currently image analysis requires new machine learning tools and specialized artificial intelligence expertise. AWS announced the availability of their new Rekognition service that makes it easier to add image analysis to your applications. The service is leveraging the learning's from the analysis of billions of Amazon prime photos. This service will allow businesses too quickly and cost effectively analyze images to identify familiar customer faces and sentiment. Immediate actions can be taken based on customer preferences to improve their experience . 

Voice Processing

With the introduction of voice assistant technology, Apple Siri, Amazon Alexa, and Google Home it has become clear customers prefer to communicate with businesses via voice rather than just a screen and keyboard. The problem is integrating voice communication into applications has required new, and complex artificial learning tools, infrastructure, and skill sets. AWS announced the availability of their voice to text service, Amazon Lex and their text to voice service, Polly. These fully managed services leverage the complex deep learning functionalities of automatic speech recognition (ASR), and natural language understanding (NLU) algorithms leveraging the massive amount of data and infrastructure supporting the Amazon Alexa products. These services will allow businesses to create a new category of speech integrated applications quickly and cost effectively. These services will accelerate the development of new customer engagement and experiences. Capital One demonstrated their use of the Lex and Polly services to create new customer engagement here.

IoT Processing

Gartner, Inc. forecasts that 6.4 billion connected things will be in use worldwide in 2016, up 30 percent from 2015, and will reach 20.8 billion by 2020. In 2016, 5.5 million new things will get connected every day. It is a huge challenge and opportunity to take full advantage of all the new data generated by these connected devices. New custom software must be developed to collect, filter, aggregate, analyze, act on, and ultimately push the data to the cloud for deeper analytics. Much of the new custom software is not run on the same limitless compute and storage capacity of the cloud. IoT software development has required specialized tools and technologies. AWS announced the Greengrass service that is designed to extend the AWS Lambda programming tool set and technology to small, simple, field-based devices. This service will simplify and accelerate new IoT software development by providing common programming tools (AWS Lambda) and technology for software running at the device, field, and cloud. Businesses collecting, acting, and analyzing data from millions and billions of devices will create new competitive advantages in product production and service efficiency. More information on AWS Greengrass is available here.

These new AWS services will allow businesses to collect, and analyze the data about their consumers faster and more deeply to better anticipate needs. Integrating image and voice processing will create new more intimate customer experiences. AWS is leading the way making these technologies readily available which will accelerate the development of new application categories in 2017. A recording of Andy Jassy's complete keynote address is available here.

IoT - Winning the IT Gold Rush

The Internet of Things (IoT) is the new IT "gold rush". IoT promises to revolutionize everything we do--the way we Live, Learn, Heal, Work, Get around, and What we eat. Every technology company is positioning new products and services to enable IoT for businesses. This is creating a lot of confusion, and unrealistic expectations. The smart business leaders I'm talking to today are leveraging the patterns of previous technology revolutions to guide their IoT strategy. Technology revolutions tend to take a decade or more to generate meaningful revenue. The challenge can be once we reach the tipping point those that are not prepared will become irrelevant quickly. The companies preparing smartly today will reap the IoT rewards of tomorrow.

I believe there are three waves of IoT adaption:

  • Wave 1 - IoT infrastructure – installing the modern compute, network connectivity, and data storage capability
  • Wave 2 - IoT applications – building the new application that will enable new products, and services leveraging the IoT infrastructure of Wave 1
  • Wave 3 – IoT enabled transformation of industries leveraging the applications and infrastructure of Wave 1 & 2

Today we are clearly in the first wave of IoT adaption. Businesses are adding IT capability for IoT workloads. Two major IT trends that I see from customers are:

  • new capability to handle the volume, variety, and velocity of IoT data
  • new data analytics capability

The smart businesses I'm working with are not building this capability as a new technology silo but instead building these new capabilities integrated with their existing IT infrastructure capability. If you look at the adaption patterns of the three main technology disruptions of the past twenty years: internet, mobile, and cloud they continued to leverage the capability and data from the previous generation. Today almost all the most successful new mobile applications are accessing existing customer relationship data. The smart businesses are adding new flash, and NVM data media to their IT infrastructure that is capable of ingesting, and processing data at 10 to 100x faster speeds to their architectures. When you have a single Wind Turbine generating 400 data points a second, a wind farm will easily overwhelm the IT infrastructure of most enterprises today. But if you incrementally add new data media like flash, and NVM technologies combining it with access to traditional product maintenance records you can start to greatly improve the output and reduce maintenance costs for your product.

The second major enterprise  infrastructure trend I am seeing for IoT is the investment in next generation data analytics capability. From a technology perspective smart businesses are gathering their data in a virtual repositories called data lakes. The type of data and minimal structure different them from traditional data warehouses where the analytics processing is often predetermined. New roles such as chief data officers, chief data analytics officer, and data scientists are being created to better understand the businesses data assets, and govern those assets. I had a CTO of a major US health care provider tell me last week he has an unlimited amount of data available but the winners will be able to mine more data, faster for actionable information. Imagine if a patient with a chronic disease like high blood pressure could be monitored 24x7x365 by a consumer cost wearable device that could collect your vital signs every minute. Using the minute by minute data your healthcare provider could be compared your information to 1000's of others to optimize your maintenance care continuously. The same device would alert your doctor immediately if your vital signs indicated the need for immediate attention. The IT capabilities including network connectivity, data processing speeds, and data science knowledge are being created right now.

I work for Dell EMC and we are focused on augmenting our products to enable the new IoT infrastructure capabilities needed. Our CTO, John Roese recently presented our strategy at IOT Solutions World Congress. During this interview at the conference he summarized Dell Technologies IoT vision to provide the new IoT infrastructure capabilities to enable the second wave of adaption.


With any new technology it is easy to get caught up in the hype and excitement of the possibilities. The smart businesses will apply the learnings of the past to be prepared for the inevitable tipping point of IoT. Smart businesses are investing to add the new IT infrastructure capabilities that are needed for the next waves of IoT adaption. The second and third waves will come faster. Business that have the necessary IoT capability and can efficiently access their existing systems of record and data will be most successful.