Regression Discontinuity Designs in Higher Ed Research

 Previously I posted a brief introduction to RD designs. Here are some applications in Higher Ed research in the areas of developmental education and financial aid:

van der Klaauw (2002). Estimating the effect of financial aid offers on college enrollment: A regression-discontinuity approach. International Economic Review.  43(4), 1249–1287.

An important problem faced by colleges and universities, that of evaluating the effect of their financial aid offers on student enrollment decisions, is complicated by the likely endogeneity of the aid offer variable in a student enrollment equation. This article shows how discontinuities in an East Coast college’s aid assignment rule can be exploited to obtain credible estimates of the aid effect without having to rely on arbitrary exclusion restrictions and functional form assumptions. Semiparametric estimates based on a regression–discontinuity (RD) approach affirm the importance of financial aid as an effective instrument in competing with other colleges for students.

Brian G. Moss  and William H. Yeaton 
Shaping Policies Related to Developmental Education: An Evaluation Using the Regression-Discontinuity Design. EDUCATIONAL EVALUATION AND POLICY ANALYSIS September 21, 2006 vol. 28 no. 3 215-229

Utilizing the regression-discontinuity research design, this article explores the effectiveness of a developmental English program in a large, multicampus community college. Routinely collected data
were extracted from existing records of a cohort of first-time college students followed for approximately 6 years (N = 1,473). Results are consistent with a conclusion that students’ participation in the program increases English academic achievement to levels similar to those of students not needing developmental coursework. The findings are also consistent with a conclusion that those students in greatest need of developmental English benefit the most from the program. This study provides an inexpensive, inferentially rigorous, program evaluation strategy that can be applied with few additional efforts to assess existing programs and to guide policy decisions.

Lasik, S.A. (2008). Evaluating developmental education programs in higher education. ASHE/Lumina Policy Brief Issue 4

A key benefi t of the regression-discontinuity design is that it effectively assesses the extent that developmental programs result in improving student retention and academic success.

Colcagno, J. C. and Long, B.T. (2008).  The Impact of Postsecondary Remediation Using a Regression Discontinuity Approach: Addressing Endogenous Sorting and Noncompliance. NCPR Working Paper.

Remedial or developmental courses are the most common policy instruments used to assist underprepared postsecondary students who are not ready for college-level coursework. However, despite its important role in higher education and its substantial costs, there is little rigorous evidence on the effectiveness of college remediation on the outcomes of students. This study uses a detailed dataset to identify the causal effect of remediation on the educational outcomes of nearly 100,000 college students in Florida, an important state that reflects broader national trends in remediation policy and student diversity. Moreover, using a Regression Discontinuity design, we discuss concerns about endogenous sorting around the policy cutoff, which poses a threat to the assumptions of the model in multiple research contexts. To address this concern, we implement methods proposed by McCrary (2008) and discuss the strengths of this approach. The results suggest math and reading remedial courses have mixed benefits. Being assigned to remediation appears to increase persistence to the second year and the total number of credits completed for students on the margin of passing out of the requirement, but it does not increase the completion of college-level credits or eventual degree completion. Taken together, the results suggest that remediation might promote early persistence in college, but it does not necessarily help students on the margin of passing the placement cutoff make long-term progress toward earning a degree.

Actually there is a nice bibliography of research related to RD designs provided by Keith Smolkowski here: 

http://homes.ori.org/~keiths/bibliography/statistics-regdisc.html

I’ve pulled out most of the references related to higher education research and education in general:

Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 86(1), 226 -244.

Seaver, W. B., & Quarton, R. J. (1976). Regression-discontinuity analysis of Dean's List effects. Journal of Educational Psychology, 68(4), 459-465.

Wong, V.C., Cook, T. D., Barnett, W.S., & Jung, K. (2008). An effectiveness-based evaluation of five state pre-k programs. Journal of Policy Analysis and Management, 27(1), 122-154

Bloom, H. S. (2012). Modern regression discontinuity analysis. Journal of Research on Educational Effectiveness, 5(1), 43-82.

Ludwig, J., & Miller, D. L. (in press). Does Head Start improve children's life chances: Evidence from a regression discontinuity approach. Quarterly Journal of Economics, 122(1), 159-208

Gorard, S., & Cook, T. D. (2007). Where does good evidence come from? International Journal of Research and Method in Education, 30(3), 307-323.

Lesik, S. (2006). Applying the regression-discontinuity design to infer causality with non-random assignment. Review Of Higher Education, 30(1), 1-19.

Marsh, H. W. (1998). Simulation study of nonequivalent group-matching and regression-discontinuity designs: Evaluations of gifted and talented programs. Journal of Experimental Education, 66(2), 163-192.

Reardon, S. F., & Robinson, J. P. (2012). Regression discontinuity designs with multiple rating-score variables. Journal of Research on Educational Effectiveness, 5(1), 83-104

 

Schochet, P. Z. (2008). Technical methods report: Statistical power for regression discontinuity designs in education evaluations (NCEE 2008-4026). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from the National Center

for Education Evaluation: http://ncee.ed.gov 

Schochet, P. Z. (2009). Statistical power for regression discontinuity designs in education evaluations. Journal Of Educational And Behavioral Statistics, 34(2), 238-266

Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J.R., Porter, J., Smith, J. (2010). Standards for regression discontinuity designs. Washington DC: U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf

Schumacker, R. E., & Mount, R. E. (2007). Regression discontinuity: Examining model misspecification. Paper presented at the 2007 Annual Meeting of the American Educational Research Association, April, 2007 Chicago,Illinois

Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309-317

Angrist, J. D., & Lavy, V. (1999). Using Maimonides' Rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 144(2), 533-576

Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F. (2008). Reading First Impact Study Final Report (NCEE 2009-4038). Washington, DC: National Center for Education Evaluation and Regional Assistance Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/pubs/20094038.asp