Saturday, October 22, 2011

Selection problem in reasoning

Employees who have chosen to join a labour union seem to make less money than their coworkers who have abstained from union membership. So why would anyone join a labour union to begin with?

J.J. Heckman
The above example illustrates sample selection bias, a typical fault in scientific reasoning that was first explicated in detail by D.B. Rubin (I am not 100% sure) and later addressed by J.J. Heckman (worthy of Nobel prize in 2000). The above reasoning is faulty because people who are likely to benefit from union membership join a union, while those unlikely to benefit decide not to join. It happens to be that in general those who earn more are less likely to benefit from the union membership.

Selection bias in academic research
The selection bias (a specific case of the broader problem of 'endogeneity') is discussed in detail in every good PhD program. Much of the advanced statistics (stuff covered after basic and time-series regression models) relates to the problem of endogeneity. Yet, selection bias remains an endemic weakness in strategic management, and probably plenty of social sciences studies.Some academics have joked that if the reviewer does not like an article and would like it to be rejected, (s)he can always complain about selection bias and endogeneity.

I recently read an article in Academy of Management Journal that I really liked, finding in effect that the actively involvement of managers and their tendency to draw in external stakeholders to discuss problems increased both the quality of resulting agreements and the resulting actions.It is a strong and enticing article. Yet, the author neglects to discuss the possibility that managers are unlikely to engage with problems or call in external stakeholders when they are thorny: managers select which problems to attend to based on their likely ability to resolve the issues. Thus, the seemingly self-evident prescription that the more managers and stakeholders engage with issues the better may be false. Indeed, it is easy to see that when problems cannot ultimately be solved, managerial engagement and the involvement of stakeholders can have high cost for the managers themselves - if not the organization.

Why are academics unable to reason correctly and to attend to the selection bias? I think there are three issues. First, it is very difficult to robustly correct for selection bias. We would have far less research done and published if we insisted on controlling for all potential selection problems. Second, the PhD education, while attending to selection problems, also creates a lot of trust into basic methodologies, both quatitative and qualitative. Researchers will always be overjoyed with any novel findings they make, It is not very enticing for us to go and try to decimate our results. Management being such a shitty practically oriented discipline that it is nearly impossible to publish studies that identify relationships and then prove them to be spurious. Finally, researchers often have a good qualitative understanding of their research subjects. When you know the managers and know how they think, you immediately know that the selection bias is not an issue. If you know it is not an issue, you may not think it is worth doing a lot of extra work to prove it conclusively.

Selection bias of managers?
To my best knowledge (not saying much), nobody has really examined selection bias in managerial reasoning. The phenomenon lies under a broader umbrella of 'superficial learning', the idea that managers learn wrong lessons from their experiences. In reality, we do not know the extent to which managers assume causality from mere correlation.The bias would seem likely: managers supposedly imitate the behaviors of their successful competitors, even though the only reason why less successful companies do not behave in the same way lies in the inability of less competitive companies to benefit from the practices.

The question is pretty significant for two reasons. First, selection bias leads to false causal attributions and thereby wrong decsisions. Second, problems resulting from selection bias can be influenced. By drawing attention to problematic causal attributions, managers can either correct their mistakes or at least approach their causal attributions and knowledge with the required scepticism.

How to study the selection bias in real life? I suppose we would require very intelligently deviced large-scale surveys. The next step would be to design laboratory experiments to investigate potential ways to mitigate biases. While neither form of research really appeals to me, I hope someone would investigate this.  

Selection bias and network centrality
When doing my PhD I had some data from a big telecoms firm to examine interpersonal networks within a big R&D unit. I found, along with the prior research, that engineers who had worked with other central engineers created inventions with greater impact within the firm. However, once I looked at the technological domains these people worked with, there was no longer any causal relationship: engineers were well connected if they worked on technologies that were crucial to the firm and the inventions of these engineers had big impact only because of the technological area they operated in. It turned out that the social ties were selected based on the work task, which also explained the apparent 'productivity'.

The data was not rich enough to "disprove" the importance of centrality in explaining 'innovative productivity', and I lost my interest in the whole domain area over time. Yet, my own observations provide a nagging feeling that many effects reported in research are significantly weaker than expected, but our social sciences are terrible in self-correcting themselves.

No comments:

Post a Comment