So much data, so little time.
In 2005, a little book called Freakonomics by William Morrow caused the world to collectively express their best Keanu Reeves impression when we all said "Whoa".
Thus the power of data was presented mainstream. Many of the conclusions presented were controversial, yet data doesn't lie. Or does it?
Since 2005 additional works have been published to refute specific claims. More data was presented with even more points of view causing us to just keep nodding and say, "yup, looks right, can't argue with that."
So what does this prove? Can big data be deterministic, meaning that it can speak for itself without human interpretation?
I think not.
Isn't the point of data interpretation? And isn't your interpretation skewed by your level of knowledge, experience and confirmation bias?
This is exactly why academic discussion is so important. We are far smarter collectively then {replace "then" with "than"?} we are on our own.
But is this a slippery slope? Are we likely to debate ourselves to death before making a decision? At some point, somebody has to do something.
Malcolm Gladwell illustrates a fascinating point about the dangers of too much data in his book "Blink". He presents empirical evidence for the powers of the subconscious to analyze intense amounts of data in a single second. You blink, and your gut tells you what to do.
This is why so often you feel as though you made your decision without really understanding why. As Keanu Reeves was so eloquently told in "The Matrix Reloaded, "you've already made your choice. Now you have to understand why you made the choice."
It is our search for why that trips us up. Our subconscious is the most advanced supercomputer on the planet telling us what to do in a single second. Yet we don't quite trust it. We spend time consciously proving through data to convince our conscious minds of what we may already intuitively know.
In fact, studies have shown that too much data just clouds the issue. We make poorer choices when we have too much data. How much is too much? Depends on the individual and depends on the problem.
Ultimately the problem is that data is never the full story. It is only just a part, and it is easy to manipulate and present conclusions one way or the other.
For example, did you know that 100% of people who eat tomatoes die?
Sounds terrifying. Better stop eating tomatoes!
Wait...what?
Adam Kolozetti