Academic journal article Air & Space Power Journal

The Big Data Imperative

Academic journal article Air & Space Power Journal

The Big Data Imperative

Article excerpt

Disclaimer: The views and opinions expressed or implied in the Journal are those of the authors and should not be construed as carrying the official sanction of the Department of Defense, Air Force, Air Education and Training Command, Air University, or other agencies or departments of the US government. This article may be reproduced in whole or in part without permission. If it is reproduced, the Air and Space Power Journal requests a courtesy line.

Big data is the subject of much discussion in the media and in the government today. It has been described as an "easy button," when combined with artificial intelligence, to reduce the human role of analysis. Some view this as a potential threat to the democratic order, and by others it is viewed as a lot of hype with few earth-shattering results to show.1 What is big data, and why is it vital to the future of the intelligence community (IC) and combined military operations?

In this article, the authors argue that the information revolution has radically changed intelligence by dramatically increasing the number and variety of intelligence collectors. Thereby the collectors create a global network of analysts and machines that facilitate the rapid sharing of data and information. This network also increases the appetite of operators for faster and more operationally relevant assessments about threats and targeting opportunities. Further, it has reshaped the threat environment by creating new centers of power and collection in the cyber domain-where adversaries can recruit members, plan strikes, and exploit both ordered and inspired attacks through online collectives. Our current manpower and resource-constrained environment-combined with these factors-necessitates new strategies for planning and executing intelligence, surveillance, and reconnaissance (ISR) operations, and investment in organizing, training, and equipping analyst Airmen with the tools to succeed in the modern information environment. Big data conceptually sits at the core of this environment and will drive our understanding of how we collect, structure, and analyze data, information, and intelligence in the future.

Cutting through the Hype-What is Big Data?

As the name implies, big data is ultimately about the gathering, storing, and processing of large volumes of data and information. Intelligence analysts will quickly point out that there is nothing new about gathering and storing large volumes of information, as it has been a central purpose of intelligence entities for centuries. Nonmilitary analysts regularly sort through large volumes of data to make quantitative assessments of complex problem sets based on tens of thousands of case observations across multiple variables. So, what makes big data new and different? The phrase first appeared in the early 2000s, when industry analyst Doug Laney defined big data as distinct from previous models by three main factors dubbed the "three Vs:"2

* Volume-The information age enables both the acquisition and storing of data and information that can be preserved and regularly accessed and analyzed on scales not seen before. Most previous databases for analysis could be contained in a single database (such as a Microsoft Excel database) with lines ranging from tens to tens of thousands of lines. Big data enables the collection of millions to billions of data points.

* Velocity-The volume of data and information is acquired at an unprecedented speed and must be dealt with promptly. Twitter, for instance, received 500 million updates (tweets) per day in 2013;3 each tweet constituting a single data point of information.

* Variety-Data and information come in numerous formats from diverse sources. In the past, the analyst or entity requiring the information could shape what was collected and how it was stored, but the combination of volume and velocity today necessitates building systems to manage and incorporate data in the form in which it is acquired; from an image to a Twitter or Facebook entry to a transcript of a conversation or speech. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.