继续质疑实验方法和访谈/调查方法中的问题

关于实验方法：更多的大牛质疑这其中的问题

Robert J. Sampson. 2010. Gold Standard Myths: Observations on the Experimental Turn in Quantitative Criminology. J Quant Criminol (2010) 26:489–500

这篇文章对许多问题讨论的非常透彻（目前，实验方法大概在三个领域里运用最多：犯罪学，教育，labor economics）

Angus Deaton and Nancy Cartwright. 2016. Understanding and Misunderstanding Random Controlled Trails (RCT). NBER working paper 22595. 原理讲的很清楚。文章很长（我也没有仔细读完）。

对实验研究报告的最严厉批评是：这些作者们明明知道自己的研究没有达到Random Controlled Trails (RCT)对Random Control的要求，但是文章照样是按达到了Random Control的要求才得以发表的。

而这样的做法，要是在自然科学领域，就算为发表而造假了！

Coppock and Donald Green (the Donald Green), 2014. Assessing the Correspondence between Experimental Results Obtained in the Lab and Field: A Review of RecentSocial Science Research, Political Science Research and Methods. DOI:http://journals.cambridge.org/abstract_S2049847014000107

D. Green是实验方法的大力推动者。但是，他们即便是比较 lab experiment和field experiment，发现对应（correlation）也只有Spearman’s r=0.73.这至少表明，某一类experiment至少有1/4（0.27）以上的东西是有问题的（考虑到p value hacking和inflation of results的话，有问题的研究结果只会更多）。而且他们的整个样本不大。再说，许多lab experiment的结果也不能重复（我此前的博文已经提到，不再赘叙）。

他们也承认其中的诸多问题，比如: “This correlation may be distorted by the ad hoc manner in which lab-field comparisons are constructed (as well as the selective manner in which results are reported and published).”

“Field experiments tend to assess the effects of a real-world intervention on those who would ordinarily encounter it. Although field experiments frequently use surveys to assess outcomes (cf. Glennerster and Takavarasha 2013), they otherwise tend not to alert subjects to the fact that they are being studied in connection with a particular intervention, and often measure outcomes unobtrusively after the intervention occurs. Situating an experiment in a field setting risks implementation problems, either because some subjects do not receive the treatment towhich they were randomly assigned or because subjects go missing before their outcomes can be measured.”

最后是一篇关于labor market中的实验方法的很长的综述文章：

Jesse Rothstein and Till von Wachter. 2016. “Social Experiments in the Labor Market,” NBER Working Paper, 22585. 这一篇文章非常长，我没有仔细看。

【当然，我对政治学、社会学、经济学中大力推行field experiment的最核心批评是（上篇博文，没敢太得罪大家，没有把这段话说出来）：能够被实验的问题，大概都是trivial的问题。（一定程度上说，这也是一个external validity的问题。）大家看D. Green的文章里提到的研究就知道了。总之，我的意思是，不要因为目前field experiment的流行，就觉得这一定是好的或者是对的方向。至少，需要保持一种谨慎的批评态度。】

最后，我还要推荐一篇质疑“微贷款（micro credit）”，特别是给妇女的“微贷款”是否真正有减少贫困的作用的文章（micro credit可是让Muhammad Yunus得了Nobel和平奖的东西呀！）。这其实也是质疑field experiment的论文结果的一篇文章。David Goodman and Jonathan Morduch. 2014. The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence, Journal of Development Studies 50 (4): 583-604.

Goodman and Morduch（2014）质疑的文章是M. M. Pitt & S. R. Khandker, 1998. ‘The impact of group-based credit on poor households in Bangladesh: Does the gender of participants matter?’, Journal of Political Economy, 106,。 而Pitt& Khandker的文章可以说是这个话题的基础文献。

基于访谈和survey的研究中需要注意的问题

两篇对访谈（以及基于访谈的survey）方法的警示（甚至是质疑），特别是有族群因素的问题（不管研究问题是否是关于族群问题）。

特别强调：（几乎）所有的field experiment都需要访谈（以及基于访谈的survey）方法来获得数据（这一点许多人忽略了）。因此，以下这两篇文章中指出的访谈（以及基于访谈的survey）研究中的问题也对我们质疑许多“实验方法”的研究中的问题很有启发。

Colin Jerolmack and Shamus Khan. 2014. Talk Is Cheap: Ethnography and the Attitudinal Fallacy, Sociological Methods & Research 43(2) 178-209.

简而言之：想法和行为有很大的gap。这篇文章接近否定这一类试图用访谈来揣摩行为的努力。

“Because meaning and action are collectively negotiated and context-dependent, we contend that self-reports of attitudes and behaviors are of limited value in explaining what people actually do because they are overly individualistic and abstracted from lived experience.” 

【也请参考我关于社会科学中的“客体”(objects)的讨论，见Shiping Tang, “Idea, Action and Outcome: The Objects and Tasks of Social Sciences，” https://fudan.academia.edu/ShipingTang 】

【这也是我们的选举预测研究不依赖民调的一个重要原因。行为在最后一刻都可以变化。】

Claire L. Adida et al. 2016. Who’s Asking? Interviewer Coethnicity Effects in African Survey Data. Comparative Political Studies. DOI: 10.1177/0010414016633487

摘要：

“(…) We find that respondents give systematically different answers to coethnic and noncoethnic interviewers across surveys in 14 African countries, but with significant variation in the degree of bias across question types and types of noncoethnic dyads, with the largest effects occurring where both the respondent and interviewer are members of ethnic groups with a history of political competition and conflict, and where the respondent or interviewer shares an ethnicity with the head of state. Our findings have practical implications for consumers of African survey data and underscore the context dependence of the social interaction that constitutes the survey experience.”

概要来说，不同的人去访谈会得到非常不同的答案，而最重要的区别是访谈的人是否是本族人，还是外族人。

顺便提一句：Adida是James Fearon和David Laitin的学生（如今是美国政治学job market上的killer combination。光是UCSD大概就有3、4个他们的学生）。