Field Experiments in Corporations

January 28, 2025

Field Experiments in Corporations

Causal inference has become a mainstream social science methodology (see here ​​for my D-Lab Blog Post on causal inference in recent political economy research). As a critical testing tool, field experiments, joining other types of experiments (e.g., survey experiments, natural experiments, quasi-experiments, etc.), are taking the momentum to catch scholarly attention. Unlike traditional lab experiments, where researchers design a controlled environment in laboratory settings, field experiments move beyond the door boundary and investigate the efficacy of certain products and policy proposals in a real-world setting.

Why would researchers strive for a natural environment if lab experiments are more realistic, more controllable, and less costly? Field experiments have their own unique strengths: when the research questions are too big in scope (e.g., the 2019 Nobel Prize laureates in Economic Sciences Abhijit Banerjee, Esther Duflo and Michael Kremer on global poverty), when the study contains real-world policy prevalence (e.g., randomized trials on the effectiveness of a social program conducted in developing countries), and when high internal validity is in need (i.e., researchers are able to minimize the impact of pre-existing differences between groups to construct robust causal relationships)

In this blog post, I first review the nascent studies in management and strategy, where corporations serve as the destination for field experiments, and introduce the interesting findings. Next, I provide a brief summary of the challenges and constraints of field experiments, and brainstorm some practical solutions to tackle the problems. The tips offered at the end of the blog aim to shed some light on how to effectively proceed with field experiments in private corporations.

Research Developments in Field Experiments: Strategy Research

“The future of social science is with private companies.” In a Harvard Business Review article, Michael Schrage underscores the importance of applying social sciences wisdom to private enterprises. Political economists and management scholars are walking on the frontiers to file field experiments with companies. The studies span a wide range of theoretical themes: censorship, sustainability, corporate behavior, and beyond.

To understand censorship among private actors in autocratic regimes, Beazer et al. (2022) reach out to Russian media firms and offer advertisements. In the email request, Russian private media companies can choose the advertisements among “preserving historical buildings,” “fighting against the repression of free speech,” and “fighting against the repression of free speech” with a boosted image of shackles and a keyboard. From the lowest to the highest, the three messages contain different levels of anti-regime sentiments. By focusing on the censorship behavior of private media firms, Beazer et al. (2022) conclude that anti-regime messaging is effective in arousing private actors’ participatory behavior in censorship.

Moreover, field experiments are influential in studies related to sustainability. For instance, Kronrod, Grinstein, and Wathieu (2012) are interested in understanding the language of environmental advertisements: would aggressive messaging be more effective in obtaining consumer compliance? They use Google AdWords to file a field experiment. Interestingly, consumers respond more actively to pushy messages when they deem the products as important, but they need more suggestive information for other goods.

Relatedly, field experiments with company participants are powerful tools to understand corporate behavior. In a well-known field experiment study in Tanzania, Berge, Bjorvatn, and Tungodden (2015) target microenterprises - small-scale businesses that often lack sufficient access to finance - and provide microenterprise owners with various assistance. Even a modest increase in both human capital (provided through business training) and financial aid (offered as grants) enables microenterprise owners to achieve better business performance in both the short and long term. However, this positive outcome does not hold when either resource is provided in isolation.

Challenges & Constraints

Admittedly, field experiments are not immune from constraints and criticisms. There are three main challenges to field experiments: low external validity, low response rate, and ethical concerns.

To begin, despite the high internal validity advantage, field experiments suffer from doubts of external validity. The harshest attack lies in “whether the study population is sufficiently similar to a target population of interest.” In other words, to what extent is the selected sample representative of the population? Especially when it comes to experiments within companies, the tested subjects (e.g., rank-and-file employees versus senior managers) and experimental sites (e.g., the type and offices of companies) are not at the full discretion of the researchers. Successful partnerships between academia and industry are usually an important prerequisite for a field experiment to be smoothly conducted. Hence, the randomization bias - “intentionally” selecting which company to reach out and test - becomes a major critique of extending a single study’s result to other similar situations.

The second challenge lies in the low response rate. According to Chatterji et al. (2016), the average response rate of private companies is approximately 15%. Moreover, the low responsiveness phenomenon is not unique to any particular political regime; to test for the cross-continent liability of foreignness theory, Chatterji et al. (2016) demonstrate that city respondents in the United States, Japan, and China have a 17.6%, 14.7%, and 15.8% response rate respectively to foreign investors. Such a response rate may risk reducing the statistical power of the field experiment, making it harder to draw definitive conclusions.

Finally, field experiment scholars are sometimes haunted by confidentiality and ethical problems. The Institutional Review Board (IRB) at each university has been critical of the protection level of human subjects involved in treatments. According to the Belmont Report, fully informed consent can be waived under specific conditions: when the benefits of the research are significant, the risks are minimal, no physical or emotional harm is inflicted, and the research cannot be conducted by other means (Bandiera, Barankay, and Rasul 2011). Phillips (2021) offers a detailed review of the ethical challenges of field experiments. For instance, she identifies a series of potential risks exposed to human subjects: physical, psychological, social, reputational, and economic harms, exclusion harms, and outcome harms. Regarding private companies as study subjects, it is essential - and highly common - for participating employees and managers to carefully evaluate the costs and benefits for both themselves and their companies before agreeing to participate. Transparent and open communications with the institutional IRB and the targeted companies have become crucial tasks for field experiment researchers.

Some Practical Solutions

Taking both the strengths and weaknesses of field experiments, how should we better deal with such a powerful research method? Below I provide some thoughts on the practical solutions to the hurdles discussed above. Although they may not serve as a panacea, researchers can at least maximize the chance of success.

First, although scholars consider good research to contain solid external validity, it is almost impossible for any study to generate perfect externally valid results (Lynch, 1982). What researchers can do is to replicate the experimental results in different contexts. For field experiments in a single company, once funding and external connections allow, researchers may try to see whether the same treatment yields the same, or similar, results across different types of companies, or across different branch offices. More importantly, the ultimate goal of good field experiments lies in theoretical advancement. Refining theories to anticipate the potential development directions of an experiment across various settings can also be advantageous (Chatterji et al., 2016; Wilke and Humphreys, 2020).

Second, to enhance firm responsiveness, researchers can polish the treatment conditions. For instance, in a field experiment via emails, it could be effective to simplify the language sent to corporate employees. Shorter messages are then more likely to be opened and read carefully. Researchers may also consider reducing the number of treatment arms. With fewer groups, each treatment and control group will have more respondents, reducing standard errors and minimizing the issue of low statistical power. 

Finally, it is vital to ensure that ethical standards are met before filing experiments within companies. To combat the concern of confidentiality, a practical solution is to delay public dissemination of results while allowing for discussion of findings in academic seminars and to submit papers for peer review. Meanwhile, being cautious, considerate, and frank with the IRB about the potential risks and benefits of the study is the key. Researchers should bear in mind that the ethical considerations should be consistent with the established research standards and protocols - this is a golden rule for all studies, not merely for field experiments.

References

  1. Bandiera, Oriana, Iwan Barankay, and Imran Rasul. 2011. “Field Experiments with Firms.” Journal of Economic Perspectives 25 (3): 63-82. 

  2. Banerjee, Abhijit Vinayak, Esther Duflo, and Michael Kremer. 2016. “The influence of randomized controlled trials on development economics research and on development policy.” In The State of Economics, The State of the World conference at the World Bank.

  3. Beazer, Quintin H., Charles D. Crabtree, Christopher J. Fariss, and Holger L. Kern. 2022. “When Do Private Actors Engage in Censorship? Evidence From a Correspondence Experiment with Russian Private Media Firms.” British Journal of Political Science 52 (4): 1790-1809. 

  4. Berge, Lars Ivar Oppedal, Kjetil Bjorvatn, and Bertil Tungodden. 2015. “Human and Financial Capital for Microenterprise Development: Evidence from a Field and Lab Experiment.” Management Science 61 (4): 707-722.

  5. Chatterji, Aaron K., Michael Findley, Nathan M. Jensen, Stephan Meier, and Daniel Nielson. 2016. “Field Experiments in Strategy Research.” Strategic Management Journal 37 (1): 116-132. 

  6. Delmas, Magali A., and J. Alberto Aragon-Correa. 2016. “Field Experiments in Corporate Sustainability Research: Testing Strategies for Behavior Change in Markets and Organizations.” Organization & Environment 29 (4): 391-400.

  7. List, John A. 2011. “Why Economists Should Conduct Field Experiments and 14 Tips for Pulling One Off.” Journal of Economic Perspectives 25 (3): 3-16. 

  8. Lynch, John G. 1982. “On the External Validity of Experiments in Consumer Research.”Journal of Consumer Research 9 (3): 225-239.

  9. Martin, Scott L., Liao, Hui, and Elizabeth M Campbell 2013. “Directive versus empowering leadership: A field experiment comparing impacts on task proficiency and proactivity.” Academy of Management Journal 56 (5): 1372-1395.

  10. Phillips, Trisha. 2021. “Ethics of Field Experiments.” Annual Review of Political Science 24 (24):277-300. 

  11. Schrage, Michael. 2015. “Why the Future of Social Science Is with Private Companies.” Harvard Business Review

  12. Wilke, Anna M., and Macartan Humphreys. 2020. “Field Experiments, Theory, and External Validity.” In The SAGE Handbook of Research Methods in Political Science and International Relations: 1007-1035. Edited by Curini, Luigi, and Robert Franzese.