The credibility revolution in empirical economics has elevated the role of experimental evidence in recent decades. While excellent practical guides to the use of randomisation as a research design are available (e.g., Duflo et al., 2007; Rothstein and von Wachter, 2017), the article by Johan Vikström distinguishes itself by offering an insightful and valuable discussion of randomised controlled trials (RCTs) applied to labour market policies in the Nordic countries. Its focus is on the actual implementation of RCTs in a real-world policy context. Using three RCTs as examples, the article emphasises that conducting them requires substantial effort, careful planning, and strong institutional motivation, and it illustrates how researchers have addressed a range of challenges. It also highlights how the rich register data available in the Nordic countries can provide not only objective outcome measures but also tools to uncover the mechanisms underlying treatment effects and to derive additional policy lessons. Finally, the article underscores the importance of combining estimated treatment effects with programme costs to assess the feasibility of an intervention.
The article discusses several complications in designing RCTs that arise from operational, ethical, and administrative difficulties in randomly assigning study populations to treatment. These include imperfect compliance, deviations from the Stable Unit Treatment Value Assumption (SUTVA), endogenously observed outcomes, and multiple treatments. As a first comment, I note that one aspect that receives relatively limited attention is the distinction between the internal and external validity of RCTs. The appeal of RCTs lies in their ability to solve the assignment problem by creating comparable treatment and control groups. While this ensures high internal validity in well-conducted RCTs, concerns may arise regarding their external validity.
RCTs frequently rely on voluntary participation and/or tendering processes, raising questions about the extent to which their findings can be generalised beyond the study population. When members of the target population are invited to participate, those who choose to do so may differ in the distribution of the treatment effect from those who decline. A similar selection concern arises in tendering processes, as providers who expect to benefit from the intervention may be more likely to submit a tender. In such cases, the effectiveness of a nationwide rollout may differ substantially from that observed in the original small-scale trial. For this reason, compulsory participation, where feasible, is strongly preferable from the perspectives of external validity and real-world effectiveness.
My second comment concerns the close, intensive collaboration required in conducting RCTs between researchers and the practitioners responsible for administering the intervention. A central insight of the article is that successful implementation depends critically on strong administrative support. Currently, when such support exists, it often relies on personal relationships between a small number of public sector employees and researchers. To improve this situation, the author calls for the development of institutional structures to support RCTs in the Nordic countries. I strongly back this proposal and would further argue for the adoption of formal quality assurance practices similar to those used in clinical trials. These typically involve independent oversight committees to safeguard data quality and trial integrity, as well as clearly defined responsibilities for the overall conduct and supervision of the intervention. The principal challenge of this proposal, however, is that it presupposes a genuine willingness to conduct rigorous assessments of the true effects of policies, and this commitment may be uneven or lacking in some parts of the public administration.
Another important issue addressed in the article concerns ethics, and the author correctly argues that RCTs are often ethically justified. In addition to the considerations discussed, I would like to add two further observations from the perspective of research ethics. First, power calculations are a necessary component of ethical study designs, as underpowered RCTs expose participants to interventions without a sufficient likelihood of producing informative results. When adequate power cannot be achieved, the study design should be revised, or the use of the RCT reconsidered. A further ethical consideration arises in the context of compulsory participation. Legislative changes frequently alter eligibility criteria for unemployment benefits and apply uniformly to affected populations, often without robust evidence of their impact. In such circumstances, trials conducted within a smaller population and with compulsory participation can be ethically justified, as credible estimates of policy effects can then be directly extrapolated to the broader population. Provided that appropriate protocols are in place, potentially harmful trials can be terminated early, and compensation mechanisms are more feasible for a limited group than for an entire population. Despite its compulsory nature, this approach is surely ethically preferable to the current practice of continuing potentially harmful policies that affect everybody who is unemployed.
Despite substantial progress in the evaluation of labour market policies, considerable gaps in our knowledge remain. The effects of specific interventions are likely to depend on broader labour market institutions, which vary significantly from country to country, and labour market policies themselves continue to evolve. These factors underscore the ongoing need for rigorous empirical evidence, including well-designed RCTs, to assess policy effectiveness across different institutional contexts. In this regard, the article offers many valuable insights, and I would strongly recommend its inclusion on reading lists for ministries, employment agencies, and researchers concerned with the evaluation of labour market policies.