The end of statistics

For nearly hundred years, every field, life-sciences, manufacturing, high-energy physics, economics, healthcare, and others relied on basic statistics and a rather crude assumption that everything follows the Normal function. There is nothing wrong with the assumption but in a regime that works on the tails, the observation that something works for the population has little practical value. In life sciences, they have been inventing mediocre therapies for over a century, as the clinicians, their regulators, and aiding statisticians have been enamored by the mighty "p-value." They have been striving to prove that the incremental average benefits delivered to a large population are a lot better than life-saving therapies for a few. In manufacturing, they have been optimizing with constraints in an attempt to save nickels and dimes. Lean, mean and mighty, their determinism has led to incorrect decisions in the presence of uncertainty. In healthcare, they have been waiting for the protocols to change based on simplistic observations of small samples. Meanwhile, half the healthcare costs in the World could be attributed to a handful of related disease states. In physics, stuffed with engineers, they have been deploying heavy steel for finding particles and hearing waves, based on basic statistical notions. Even with that, they will be the first to admit that they do not yet know 94% of it. In economics, they have been inventing theories based on regression and even winning Nobel prizes but it is unclear if they are creating insights. Some of them ventured into even making money and some have failed spectacularly as would have been predicted by their own theories. Overall, if one can write down an equation for a process, it is symptomatic of the fact that she has not understood it. The practitioners, who seem to cling to the past are being rendered less effective in the presence of those who look forward.

A generation seems to have wasted their time adhering to basic principles laid out a century ago. Lately, statistics have been made sexier by better naming - now called, "Machine Learning." One has to admit it does sound a lot better, but has anything changed? In a world full of practicing scientists, who have been trained to make equations for everything, we are approaching a significant discontinuity. Machines are certainly marching forward but not because they know statistics but because they do not. Such is the state of affairs that a systematic education delivered by the greatest institutions in the world prepares the next generation to fail with high certainty. Meanwhile, machines can see, hear and make decisions in the presence of uncertainty. As we hunt for fossils to establish our own identity in a process that seems to have taken a long time, machines with no emotions and even less historical baggage, rise. Are humans being rendered irrelevant? As the greatest living physicist warns of ETs, as the world's richest and powerful worry about AI, and as the most powerful man on Earth worry about if his hair is falling straight, we have arrived at the precipice of a great discontinuity.

As they moved out of their homeland in Africa, humans must have made important calculations based on uncertainty. As they descended from the trees into the African Savannah, a few million years prior, they knew the regime was shifting. With dangers all around them, mighty beasts who could maul them in a single swipe, they made decisions based on uncertainty. Their initial journeys into the Middle East and South Asia, closely followed by those who went a bit North, seem to have provided a level of safety. They advanced culture and boredom, the latter most important for the development of human psyche. As the caves in Southern France prove, they could certainly rise above determinism and engineering, very early in their progression.

The regime is shifting again - the opponents are not as gentle as the Neanderthals. Machines are brutal and they are immensely capable. Humans, the victors of past conflicts, are starting from a position of a great disadvantage because of their education of the past. The end of statistics, a figment of the imagination of the most recent generation, is very near.