Wednesday, March 9, 2011

Statistics are dangerous tools in the hands of inexpert or the biased

Was working on this problem for a friend and came across the thought of identifying what errors are grossed today and maybe some root causes of these issues:

Overestimation of causality: Most of the statistics text is mis-informed to be built on the principle of causality, which people take for granted. While clearly that is not the case - in simple words "they see elephants in the clouds instead of understanding that they are in fact randomly shaped clouds that appear to our eyes as elephants". This maturity would only come to a trained statistician, else everything would be oversimplified into patterns and normality.

Counter-intuitive distributions: This is antonym of the first one (in limited sense). Not everything is normal distribution and not everything can be fit to a bell curve. This premise is hardly checked before building any hypothesis.

Lack of understanding of characteristics of distributions: Seldom there are thoughts and efforts to see symptoms like(talking of commonly understood normal distribution) long tail, fat tail, kurtosis, skewness, etc. These are important yet forgotten episodes of analysis, for a simple fact that most amateurs are ignorant of how to deal with them.

Over dependency on chance and randomness: The base of probability is chance p(n), which assumes that the final output cannot be understood but probability can be identified with confidence level. There are other theories that build on deterministic models rather than chance models - chaos theory, lorentz attractor, relativity- unfortunately, due to their mathematical complexity they have been left for the work of physics and mathematics rather than we appreciating their objectivity in our "false world of randomness". There are cases where deterministic models are more appropriate than the random models.

Reducing complexity: The aim should be to simplify the problems rather than build more dynamics and complexity, but this cannot be achieved at the cost of neglecting vitality and objectivity of the outcome. This is also a critical area left unexplored.

While, I have tried to put my thoughts together for you to see which path you would want to develop, you would also observe that they might be contradicting (which is not wrong as we do not know what seems better). I feel all these aspects are grossly missed in building models and hypothesis. There are examples of VaR, financial exponentials and derivatives, inaccurate assessment of impact of interest rate revisions in economy, lack of accounting for the multiplier effect, oversimplification of regression in subsidy model, panic as observed by behaviorial economics, etc. that haunt us time and again.

There is a calling to understand how the best of works in physics, economics, mathematics, statistics, psychology can be cross leveraged to find new means of unified theory. I firmly believe - God does not play dice!"

Would appreciate your viewpoints and we could build the discussion here!


1 comment:

Jess Malfavón said...

this is the very first entry i read.. i really liked it =-D