Abstract: We investigate the use of large language models (LLMs) to simulate human responses to survey questions, and perform uncertainty quantification to gain reliable insights. Our approach converts imperfect, LLM-simulated responses into confidence sets for population parameters of human responses, addressing the distribution shift between the simulated and real populations. A key innovation lies in determining the optimal number of simulated responses: too many produce overly narrow confidence sets with poor coverage, while too few yield excessively loose estimates. To resolve this, our method adaptively selects the simulation sample size, ensuring valid average-case coverage guarantees. It is broadly applicable to any LLM, irrespective of its fidelity, and any procedure for constructing confidence sets. Additionally, the selected sample size quantifies the degree of misalignment between the LLM and the target human population. We illustrate our method on real datasets and LLMs. 

Abstract: Multi-site/context studies have become popular strategies to address the most common and challenging external validity concerns about contexts. Under such studies, scholars conduct causal studies in each site and evaluate whether findings generalize across sites. Despite the potential, there has been little guidance on the fundamental research design question—how should we select sites for external validity? Existing approaches have challenges: random sampling of sites is often infeasible, while the current practice of purposive sampling is suboptimal without statistical guarantees. We propose synthetic purposive sampling (SPS), which optimally selects diverse sites for external validity. SPS combines ideas of purposive sampling and the synthetic control method—it selects diverse sites such that non-selected sites are well approximated by the weighted average of the selected sites. We illustrate its general applicability using both experimental and observational studies. Overall, this paper offers a new statistical foundation to design multi-site studies for external validity.

Abstract: We study the consequences of affirmative action in centralized college admis- sions systems. We develop an empirical framework to examine the effects of a large-scale program in Brazil that required all federal institutions to reserve half their seats for socioeconomically and racially marginalized groups. By exploiting admissions cutoffs, we find that marginally benefited students are more likely to attend college and are enrolled at higher-quality degrees four years later. Mean- while, there are no observed impacts for marginally displaced non-targeted stu- dents. To study the effects of larger changes in affirmative action, we estimate a joint model of school choices and potential outcomes. We find that the policy has impacts on college attendance and persistence that imply a virtually one-to- one income transfer from the non-targeted to the targeted group. These findings indicate that introducing affirmative action can increase equity without affecting efficiency.

Abstract: Two ongoing problems in experimental research are (a) more credible ideas for addressing social problems are generated than can be experimentally tested and (b) treatment effects estimated in one setting may not necessarily apply to other settings. My talk will discuss the potential of forecasting tournaments as an additional tool for solving these twin problems. In a completed study, we evaluate whether experts or laypeople can accurately forecast the efficacy of interventions to strengthen Americans’ democratic attitudes, and thus the potential of forecasters to identify the most promising interventions to test. All forecasts performed better than chance, but experts outperformed the lay public, and academic v. practitioner experts differed in their sensitivity and specificity. Hence, depending on the relative importance of avoiding false-positive vs. false-negative forecasts, decision-makers may prefer different experts. In an ongoing study, I investigate if lay people can accurately forecast how the effects of RCT-tested educational interventions generalize to their specific school districts. I also plan to benchmark the accuracy of these forecasts against those from various large language models, and whether inviting the public to participate in forecasting improves community support for evidence-based policymaking.

Abstract: Political spending is at an all-time high. It has skyrocketed from $3.1 billion in 2000 to $15.9 billion in 2024—a 416% increase. This increase has inspired campaigns to experiment with new types of political ads. In this research, we investigate a novel form of political advertising, which we refer to as “disloyalty ads.” Disloyalty ads are ads in which a candidate disagrees with their party on a political issue. How do people react to the use of Disloyalty Ads? Under what conditions do they boost candidate support? We investigate candidate-, voter-, and issue-level variation that predicts the success of this form of advertising. For this talk, I will focus on issue-level variation. Candidates using disloyalty ads can challenge their party on issues associated with the in-party (e.g., a Democrat challenging expanding LGBTQ+ rights). Alternatively, candidates can align themselves with the opposing party on issues associated with the out-party (e.g., a Democrat supporting limited government). Which approach improves candidate support? Across four studies (N = 5,142) using a new method that combines LLM embeddings and clustering algorithms, we find that candidates are better off signaling disloyalty on issues associated with the out-party.

Abstract: Democracy is in a global crisis, which technology has not so far helped us overcome. Large language models provide a new tool for finding answers to specific questions and summarizing complex information. However, the monolithic perspective on reality that chatbots provide – tuned by technology corporations – raises questions about their role in liberal democracy. I will argue that we can use AI technology to capture and map a plurality of perspectives and scale a classical element of participatory democracy: citizen assemblies. Citizen assemblies are deliberative forums of randomly selected citizens brought together to discuss a particular issue and share their opinions. The goal is for the opinions of the participants to impact the political process, e.g.,  through recommendations to representatives of the government. In recent years, citizen assemblies have gained some traction worldwide, particularly in Europe and North America. For instance, the French Citizens’ Convention for Climate (2019-2020) brought together 150 citizens to propose measures for reducing greenhouse gas emissions. Similarly, the Irish Citizens’ Assembly (2016-2018) played an important role in shaping public opinion and government policy on issues like abortion and same-sex marriage. AI technology can help us (1) scale citizen assemblies affordably to thousands or millions of participants and (2) map the landscape of nuanced opinions in a transparent and verifiable fashion. Assemblies can be recorded and transcribed automatically, and the variety of positions summarized using language models. Novel methods combining language models and multivariate analyses can then capture the complete space of opinions as a graph whose nodes represent opinions at a range of levels of abstraction, from broad to nuanced. People (including but not limited to the participants of the assemblies) can traverse the graph online, attach ratings of their degree of belief in the opinions, and add opinions not yet captured. Implementing this project at Columbia would require a broad collaboration among faculty spanning the humanities, the social sciences, the natural sciences, and engineering. The project would leverage modern AI to help make good on the promise of liberal democracy.