Wednesday, May 22, 2013

Regression Discontinuity Designs in Higher Ed Research

 Previously I posted a brief introduction to RD designs. Here are some applications in Higher Ed research in the areas of developmental education and financial aid:

van der Klaauw (2002). Estimating the effect of financial aid offers on college enrollment: A regression-discontinuity approach. International Economic Review.  43(4), 1249–1287.

An important problem faced by colleges and universities, that of evaluating the effect of their financial aid offers on student enrollment decisions, is complicated by the likely endogeneity of the aid offer variable in a student enrollment equation. This article shows how discontinuities in an East Coast college’s aid assignment rule can be exploited to obtain credible estimates of the aid effect without having to rely on arbitrary exclusion restrictions and functional form assumptions. Semiparametric estimates based on a regression–discontinuity (RD) approach affirm the importance of financial aid as an effective instrument in competing with other colleges for students.

Brian G. Moss  and William H. Yeaton 
Shaping Policies Related to Developmental Education: An Evaluation Using the Regression-Discontinuity Design. EDUCATIONAL EVALUATION AND POLICY ANALYSIS September 21, 2006 vol. 28 no. 3 215-229

Utilizing the regression-discontinuity research design, this article explores the effectiveness of a developmental English program in a large, multicampus community college. Routinely collected data
were extracted from existing records of a cohort of first-time college students followed for approximately 6 years (N = 1,473). Results are consistent with a conclusion that students’ participation in the program increases English academic achievement to levels similar to those of students not needing developmental coursework. The findings are also consistent with a conclusion that those students in greatest need of developmental English benefit the most from the program. This study provides an inexpensive, inferentially rigorous, program evaluation strategy that can be applied with few additional efforts to assess existing programs and to guide policy decisions.


Lasik, S.A. (2008). Evaluating developmental education programs in higher education. ASHE/Lumina Policy Brief Issue 4

A key benefi t of the regression-discontinuity design is that it effectively assesses the extent that developmental programs result in improving student retention and academic success.

Colcagno, J. C. and Long, B.T. (2008).  The Impact of Postsecondary Remediation Using a Regression Discontinuity Approach: Addressing Endogenous Sorting and Noncompliance. NCPR Working Paper.

Remedial or developmental courses are the most common policy instruments used to assist underprepared postsecondary students who are not ready for college-level coursework. However, despite its important role in higher education and its substantial costs, there is little rigorous evidence on the effectiveness of college remediation on the outcomes of students. This study uses a detailed dataset to identify the causal effect of remediation on the educational outcomes of nearly 100,000 college students in Florida, an important state that reflects broader national trends in remediation policy and student diversity. Moreover, using a Regression Discontinuity design, we discuss concerns about endogenous sorting around the policy cutoff, which poses a threat to the assumptions of the model in multiple research contexts. To address this concern, we implement methods proposed by McCrary (2008) and discuss the strengths of this approach. The results suggest math and reading remedial courses have mixed benefits. Being assigned to remediation appears to increase persistence to the second year and the total number of credits completed for students on the margin of passing out of the requirement, but it does not increase the completion of college-level credits or eventual degree completion. Taken together, the results suggest that remediation might promote early persistence in college, but it does not necessarily help students on the margin of passing the placement cutoff make long-term progress toward earning a degree.


Actually there is a nice bibliography of research related to RD designs provided by Keith Smolkowski here: 

I’ve pulled out most of the references related to higher education research and education in general:

Jacob, B. A., & Lefgren, L. (2004). Remedial education and student achievement: A regression-discontinuity analysis. Review of Economics and Statistics, 86(1), 226 -244.

Seaver, W. B., & Quarton, R. J. (1976). Regression-discontinuity analysis of Dean's List effects. Journal of Educational Psychology, 68(4), 459-465.

Wong, V.C., Cook, T. D., Barnett, W.S., & Jung, K. (2008). An effectiveness-based evaluation of five state pre-k programs. Journal of Policy Analysis and Management, 27(1), 122-154

Bloom, H. S. (2012). Modern regression discontinuity analysis. Journal of Research on Educational Effectiveness, 5(1), 43-82.

Ludwig, J., & Miller, D. L. (in press). Does Head Start improve children's life chances: Evidence from a regression discontinuity approach. Quarterly Journal of Economics, 122(1), 159-208

Gorard, S., & Cook, T. D. (2007). Where does good evidence come from? International Journal of Research and Method in Education, 30(3), 307-323.

Lesik, S. (2006). Applying the regression-discontinuity design to infer causality with non-random assignment. Review Of Higher Education, 30(1), 1-19.

Marsh, H. W. (1998). Simulation study of nonequivalent group-matching and regression-discontinuity designs: Evaluations of gifted and talented programs. Journal of Experimental Education, 66(2), 163-192.

Reardon, S. F., & Robinson, J. P. (2012). Regression discontinuity designs with multiple rating-score variables. Journal of Research on Educational Effectiveness, 5(1), 83-104
 
Schochet, P. Z. (2008). Technical methods report: Statistical power for regression discontinuity designs in education evaluations (NCEE 2008-4026). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from the National Center
for Education Evaluation: http://ncee.ed.gov 

Schochet, P. Z. (2009). Statistical power for regression discontinuity designs in education evaluations. Journal Of Educational And Behavioral Statistics, 34(2), 238-266

Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J.R., Porter, J., Smith, J. (2010). Standards for regression discontinuity designs. Washington DC: U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf

Schumacker, R. E., & Mount, R. E. (2007). Regression discontinuity: Examining model misspecification. Paper presented at the 2007 Annual Meeting of the American Educational Research Association, April, 2007 Chicago,Illinois

Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational Psychology, 51(6), 309-317

Angrist, J. D., & Lavy, V. (1999). Using Maimonides' Rule to estimate the effect of class size on scholastic achievement. Quarterly Journal of Economics, 144(2), 533-576

Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F. (2008). Reading First Impact Study Final Report (NCEE 2009-4038). Washington, DC: National Center for Education Evaluation and Regional Assistance Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/pubs/20094038.asp 

Regression Discontinuity Designs

Suppose a policy or intervention is implemented or a treatment is applied based on arbitrary values of some observed covariate value or values X0. If there is some positive relationship between ‘X’ and the outcome ‘Y’ then how do we know if a treatment applied to subjects where X > X0 isn’t biased since subjects with higher values of X are more likely to exhibit higher levels of the outcome variable Y anyway?  Is it valid to make comparisons of observed outcomes (Y) between groups with differing values of (X)?  One solution would be to implement matched comparisons between groups with similar values of covariates.  Regression discontinuity designs allow us to compare differences between groups in the neighborhood of the cutoff value X0 giving us unbiased estimates of treatment effects.





  • Treatment effects can be characterized by a change in intercept or main effect at the discontinuity.
  •  Treatment assignment is equivalent to random assignment within the neighborhood of the cutoff   (Lee & Lemieux,2010).
  • More complicated functional forms may be estimated:

Y = f(x) +ρ D + e where f(x) may be a pth order polynomial
  • Comparisons of outcomes in the neighborhood of X0 provide estimates of the treatment effect  ρ that does not depend on an exactly correct specification of the functional form of E[Y|X] (Angrist &Pischke, 2009)
  •  Even more complicated methods including local linear regression may be implemented

The above illustrates only one potential visualization of RD designs.  As illustrated below, treatment effects  can be visualized as discontinuities  or changes in either the intercept or slope or both at the cutoff X0





Application:

In Shaping Policies Related to Developmental Education: An Evaluation Using the Regression-Discontinuity Design,  the authors use RD design to assess the impact of developmental education on student success in subsequent level English courses :



They find that ‘students’ participation in the program increases English academic achievement to levels similar to those of students not needing developmental coursework.’ Note in this case, the treatment (developmental course work) is applied where X < X0 = 85, vs. where X > X0 in the cases I presented above. The discontinuity/treatment effect in this case is represented by a change in slope/interaction at the cutoff.

References:

Brian G. Moss  and William H. Yeaton 
Shaping Policies Related to Developmental Education: An Evaluation Using the Regression-Discontinuity Design. EDUCATIONAL EVALUATION AND POLICY ANALYSIS September 21, 2006 vol. 28 no. 3 215-229

Imbens, Guido W. & Lemieux, Thomas, 2008. "Regression discontinuity designs: A guide to practice," Journal of Econometrics, Elsevier, vol. 142(2), pages 615-635, February.

Regression Discontinuity Designs in Economics
David S. Lee and Thomas Lemieux.
 Journal of Economic Literature 48 (June 2010)281-355

REGRESSION DISCONTINUITY
PATRICIA BAUMER

Mostly Harmless Econometrics. Angrist & Pischke. 2009.

Friday, April 26, 2013

SAS Global Forum 2013 Paper 144-2013: SAS IML Worskshop

I didn't realize until now that the hands on workshops also had accompanying papers! ( I also just noticed this year  that the same was true for the posters as well).

This paper is a great intro to SAS IML. (see my other posts with statistical programming applications in social network analysis, text mining, and maximum likelihood estimation here )

Paper 144-2013
Getting Started with the SAS/IML® Language
Rick Wicklin, SAS Institute Inc.

ABSTRACT

Do you need a statistic that is not computed by any SAS® procedure? Reach for the SAS/IML® language! Many statistics are naturally expressed in terms of matrices and vectors. For these, you need a matrix language. This paper introduces the SAS/IML language to SAS programmers who are familiar with elementary linear algebra. The focus is on statements that create and manipulate matrices, read and write data sets, and control the program flow. The paper demonstrates how to write user-defined functions, interact with other SAS procedures, and recognize efficient programming techniques.

Link: http://support.sas.com/resources/papers/proceedings13/144-2013.pdf