The power of 'N=all'

A lucid explanation of the phenomenon of Big Data and how it impacts lives

Ubiquitous networking enabled by advances in Internet technologies, the pervasive connectivity of all devices and the new generation of applications that enable peer-to-peer communication across devices and people have all resulted in the generation of petabytes of data per day, thus coining a new paradigm referred to as "Big Data". Though there are a myriad books available on Big Data, which closely look at the architecture, analytics and algorithms, the authors of this one clearly articulate the changes brought about by Big Data in a lucid, easy-to-read yet thorough, intense analysis.

The paradigm shift from the age-old sample-based statistical analysis of data to "N=all" is a very powerful concept that the authors explain in different chapters of the book. Though it is not good for statisticians who use sophisticated empirical tools and methodologies to analyse sample data to predict the behaviour of the population and draw causal relations, the authors clearly indicate that Big Data enables us to correlate (instead of establishing cause-effect) different factors and explain what a phenomenon is and not why it happens. The answer to what enables businesses to reap the benefits of the Big Data, as is evident in the case of Amazon's recommendation system that "knows what drives clicks of a book" instead of trying to unravel the mysteries of "why a user clicked to buy a book".

The authors explain why the "messiness" of Big Data does not cause any harm to the analysis because the N=all fixes it; in fact, it is even better than leaning in favour of fewer and exact data, especially for many tasks in business and life for which an ultra-precise answer is not needed. A faster and synergistic answer is the need of the hour, the authors claim through numerous examples.

Big Data has come into the limelight today only owing to the fact that the world is transitioning from analog to digital. Digitisation of data has enabled categorisation and quantification that in turn enables analysis. The authors clearly explain the difference between "digitisation" and "datafication" through Google's digital book project. Numerous examples from the field of healthcare and e-commerce are given to drive home the power of datafication, and hence the relevance of Big Data. The book also gives numerous examples of how firms differ (the oft-repeated one being a comparison between Amazon and Google) in extracting value out of data. Issues such as the primary and secondary uses of data, the extensibility of data, the depreciating value of data and the Big Data value chain are brought out succinctly. The ominous prediction of the authors is that "the experts are dead; long live Big Data". The biggest impact of Big Data for organisations is that decisions will be based increasingly on data, and less and less on human judgment and an expert's prior experience!

Obviously, the book also takes a closer look at the risks of Big Data including privacy violations, preempting punishments, and surrender of freedom to Big Data owners and analysts. The data dictatorship story of Robert McNamara, the US secretary of defence who was president of Ford, is presented to illustrate these issues in the era of Big Data.

In the last chapter, the books makes predictions with data taking centrestage in our day-to-day lives, with new tools from faster processes and more memory to smarter software and algorithms crunching the data and drawing correlations. In this future, Big Data will provide solutions not only for trivial problems but also for global problems including improving the lives of the world's poor, finally leading to even bigger data especially in scientific research. The book ends with a touching sentence: "…we must use this Big Data tool with a generous degree of humility...and humanity".

For those who are interested in knowing who are the producers and users of Big Data, this book is a rich resource. Examples range from the Googles and Amazons to Decide.com and Oren Etzioni's Farecast to experiments at MIT and Harvard.

This is an extremely powerful book that explains Big Data through hundreds of examples and associated reasoning. It is highly recommended for anyone who is curious about the emerging phenomenon of Big Data.

The reviewer is with Sasken Communication Technologies

BIG DATA
A Revolution That Will Transform How We Live, Work, and Think
Viktor Mayer-Schonberger and Kenneth Cukier
Hatchette; 242 pages