Sandy Pentland believes that the power of “Big Data” is that it is information about people’s behaviour instead of information about their beliefs. It’s not about the things which are posted on Facebook, and it’s not about searches on Google, which is what most people think, and it’s not data from internal company processes. Pentland’s “Big Data” comes from things like the location of your cell phone or credit card as you move around in the world, and the many traces you leave in time and space of your personal choices and actions.
With Big Data traditional methods of system building are of limited use. The data is so big that any question you ask about it will usually have a statistically significant answer. This means, strangely, that the scientific method as we normally use it no longer works, because almost everything is significant. As a consequence, the normal laboratory-based question-and-answering process, the method that we have used to build systems for centuries, falls apart.