Many interventions occur in settings where treatments are applied to groups. For example, a math intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are non-randomly allocated, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed a form of matching, known as multilevel matching, that is designed for contexts where treatments are clustered. In this article, we provide a tutorial on how to analyze clustered treatment using multilevel matching. We use a real data application to explain the full set of steps for the analysis of a clustered observational study.

Policy makers periodically consider using student assignment policies to improve educational outcomes by altering the socio-economic and academic skill composition of schools. We exploit the quasi-random reassignment of students across schools in the Wake County Public School System to estimate the academic and behavioral effects of being reassigned to a different school and, separately, of shifts in peer characteristics. We rule out all but substantively small effects of transitioning to a different school as a result of reassignment on test scores, course grades and chronic absenteeism. In contrast, increasing the achievement levels of students’ peers improves students’ math and ELA test scores but harms their ELA course grades. Test score benefits accrue primarily to students from higher-income families, though students with lower family income or lower prior performance still benefit. Our results suggest that student assignment policies that relocate students to avoid the over-concentration of lower-achieving students or those from lower-income families can accomplish equity goals (despite important caveats), although these reassignments may reduce achievement for students from higher-income backgrounds.

Improving teacher selection is an important strategy for strengthening the quality of the teacher workforce. As districts adopt commercial teacher screening tools, evidence is needed to understand these tools’ predictive validity. We examine the relationship between Frontline Education’s TeacherFit instrument and newly hired teachers’ outcomes. We find that a 1 SD increase on an index of TeacherFit scores is associated with a 0.06 SD increase in evaluation scores. However, we also find evidence that teachers with higher TeacherFit scores are more likely to leave their hiring schools the following year. Our results suggest that TeacherFit is not necessarily a substitute for more rigorous screening processes that are conducted by human resources officials, such as those documented in recent studies.

Many public school diversity efforts rely on reassigning students from one school to another. While opponents of such efforts articulate concerns about the consequences of reassignments for students’ educational experiences, little evidence exists regarding these effects, particularly in contemporary policy contexts. Using an event study design, we leverage data from an innovative socioeconomic school desegregation plan to estimate the effects of reassignment on reassigned students’ achievement, attendance, and exposure to exclusionary discipline. Between 2000 and 2010, North Carolina’s Wake County Public School System (WCPSS) reassigned approximately 25 percent of students with the goal of creating socioeconomically diverse schools. Although WCPSS’s controlled school choice policy provided opportunities for reassigned students to opt out of their newly reassigned schools, our analysis indicates that reassigned students typically attended their newly reassigned schools. We find that reassignment modestly boosts reassigned students’ math achievement, reduces reassigned students’ rate of suspension, and has no offsetting negative consequences on other outcomes. Exploratory analyses suggest that the effects of reassignment do not meaningfully vary by student characteristics or school choice decisions. The results suggest that carefully designed school assignment policies can improve school diversity without imposing academic or disciplinary costs on reassigned students.

Many interventions in education occur in settings where treatments are applied to groups. For example, a reading intervention may be implemented for all students in some schools and withheld from students in other schools. When such treatments are nonrandomly allocated, outcomes across the treated and control groups may differ due to the treatment or due to baseline differences between groups. When this is the case, researchers can use statistical adjustment to make treated and control groups similar in terms of observed characteristics. Recent work in statistics has developed matching methods designed for contexts where treatments are clustered. This form of matching, known as multilevel matching, may be well suited to many education applications where treatments are assigned to schools. In this article, we provide an extensive evaluation of multilevel matching and compare it to multilevel regression modeling. We evaluate multilevel matching methods in two ways. First, we use these matching methods in a within-study comparison design and attempt to recover treatment effect estimates from three clustered randomized trials. Second, we conduct a simulation study. We find evidence that generally favors an analytic approach to statistical adjustment that combines multilevel matching with regression adjustment. We conclude with an empirical application.

The American Academy of Pediatrics recommends that U.S. secondary schools begin after 8:30 a.m. to better align with the circadian rhythms of adolescents. Yet due to economic and logistic considerations, the vast majority of high schools begin the school day considerably earlier. We leverage a quasi-natural experiment in which five comprehensive high schools in one of the nation’s largest school systems moved start times forty minutes earlier to better coordinate with earlier-start high schools. Here, disruption effects should exacerbate any harmful consequences. We report on the effect of earlier start times on a broad range of outcomes, including mandatory ACT test scores, absenteeism, on-time progress in high school, and college-going. While we fail to find evidence of harmful effects on test scores, we do see a rise in absenteeism and tardiness rates, as well as higher rates of dropping out of high school. These results suggest that the harmful effects of early start times may not be well captured by considering test scores alone.

Clustered observational studies (COSs) are a critical analytic tool for educational effectiveness research. We present a design framework for the development and critique of COSs. The framework is built on the counterfactual model for causal inference and promotes the concept of designing COSs that emulate the targeted randomized trial that would have been conducted were it feasible. We emphasize the key role of understanding the assignment mechanism to study design. We review methods for statistical adjustment and highlight a recently developed form of matching designed specifically for COSs. We review how regression models can be profitably combined with matching and note best practice for estimates of statistical uncertainty. Finally, we review how sensitivity analyses can determine whether conclusions are sensitive to bias from potential unobserved confounders. We demonstrate concepts with an evaluation of a summer school reading intervention in Wake County, North Carolina.

This paper examines curricular acceleration in mathematics during elementary school using administrative data from a large, diverse school district that recently implemented a targeted, test-based acceleration policy. We first characterize access to advanced math and then estimate effects of acceleration in math on measures of short-run academic achievement as well as non-test-score measures of grit, engagement with schoolwork, future plans, and continued participation in the accelerated track. Experiences and effects of math acceleration differ markedly for girls and boys. Girls are less likely to be nominated for math acceleration and perform worse on the qualifying test, relative to boys with equivalent baseline performance. We find negative effects of acceleration on short-run retention of math knowledge for girls, but no such performance decay for boys. After initial exposure to accelerated math, girls are less likely than boys to appear in the accelerated track during late elementary school and at the start of middle school.

This article explores the origins of youth engagement in school, community and democracy. Specifically, it considers the role of psychosocial or non-cognitive abilities, like grit or perseverance. Using a novel original large-scale longitudinal survey of students linked to school administrative records and a variety of modeling techniques—including sibling, twin and individual fixed effects—the study finds that psychosocial abilities are a strong predictor of youth civic engagement. Gritty students miss less class time and are more engaged in their schools, are more politically efficacious, are more likely to intend to vote when they become eligible, and volunteer more. Our work highlights the value of psychosocial attributes in the political socialization of young people.

Career academies serve an increasingly wide range of students. This paper examines the contemporary profile of students entering career academies in a large, diverse school district and estimates causal effects of participation in one of the district’s well-regarded academies on a range of high school and college outcomes. Exploiting the lottery-based admissions process of this technology-focused academy, we find that academy enrollment increases the likelihood of high school graduation by about 8 percentage points and boosts rates of college enrollment for males but not females. Analysis of intermediate outcomes suggests that effects on attendance and industry-relevant certification at least partially mediate the overall high school graduation effect.

In the wake of political and legal challenges facing race-based integration, districts have turned to socioeconomic integration initiatives in an attempt to achieve greater racial balance across schools. Empirically, the extent to which these initiatives generate such balance is an open question. In this article, we leverage the school assignment system that the Wake County Public School System employed throughout the 2000s to provide evidence on this issue. Although our results show that Wake County Public School System’s socioeconomic-based assignment policy had negligible effects on average levels of segregation across the district, it substantially reduced racial segregation for students who would have attended majority-minority schools under a residence-based assignment policy. The policy also exposed these students to peers with different racial/ethnic backgrounds, higher mean achievement levels, and more advantaged neighborhood contexts. We explore how residential context and details of the policy interacted to produce this pattern of effects and close the article by discussing the implications of our results for research and policy.

This paper estimates the effects of earning industry-recognized certifications (IRCs) in high school on downstream outcomes. IRCs are a form of alternative education credential issued by industry groups or corporations to individuals seeking to acquire knowledge or skills in a particular sector. This credential type has grown rapidly among high school students as school systems incorporate them into accountability systems and students respond to labor market demand for skills. Since IRCs are awarded on the basis of an objective passing score, students who just barely fail or barely pass are likely to differ only in their likelihood of earning a certification. I thus use regression discontinuity design to estimate the signaling effect of earning a certification on post-high-school intentions, postsecondary enrollment, and earnings. Conditional on first exam attempts, IRC earners are more likely to express their intent to attend a four-year-college and are just as likely to actually enroll—effects that operate through college readiness indicators and not career and technical education. However, within seven years following the first IRC attempt, students on the margin of passing earn no more than their counterparts. The results suggest that for marginal examinees, earning an IRC may represent a signal that boosts interest and enrollment in four-year college, but these effects do not translate into higher short-term earnings.

Magnet schools provide innovative curricula designed to attract students from other schools within a school district, typically with the joint goals of diversifying enrollment and boosting achievement. Measuring the impact of attending a magnet school is challenging because students choose to apply and schools have priorities over types of students. Moreover, magnet schools may influence non-cognitive skill formation that is not well-reflected in test scores. This study estimates the causal impact of attending a magnet school on student outcomes by leveraging exogenous variation arising from tie breakers embedded in a centralized school assignment mechanism. Using a rich set of administrative data from a large school district, we find suggestive evidence that attending a magnet school led to higher performance in mathematics and non-language immersion magnet schools also increased students’ reading scores. Student engagement was significantly higher, as measured through absenteeism and on-time progress rates. Further, students were significantly less likely to change schools when attending a magnet. These results provide robust evidence that magnet school—a typically understudied school choice option—can benefit student learning and increase student engagement while enabling the system to achieve its goals of promoting racial and socioeconomic balance through school choice.

Disparities in gifted representation across demographic subgroups represents a large and persistent challenge in U.S. public schools. In this paper, we measure the impacts of a school-wide curricular intervention designed to address such disparities. We implemented Nurturing for a Bright Tomorrow (NBT) as a cluster randomized trial across elementary schools with the low gifted identification rates in one of the nation?s largest school systems. NBT did not boost formal gifted identification or math achievement in the early elementary grades. It did increase reading achievement in select cohorts and broadly improved performance on a gifted identification measure that assesses nonverbal abilities distinct from those captured by more commonly used screeners. These impacts were driven by Hispanic and female students. Results suggest that policymakers consider a more diverse battery of qualifying exams to narrow disparity gaps in gifted representation and carefully weigh tradeoffs between universal interventions like NBT and more targeted approaches.

We study the effects of informal social interactions on academic achievement and behavior using idiosyncratic variation in peer groups stemming from changes in bus routes across elementary, middle, and high school. In early grades, a one standard-deviation change in the value-added of same-grade bus peers corresponds to a 0.01 SD change in academic performance and a 0.03 SD change in behavior; by high school, these magnitudes grow to 0.04 SD and 0.06 SD. These findings suggest that student interactions outside the classroom—especially in adolescence—may be an important factor in the education production function.