PRESS RELEASE:

A Message From the National Charter School Research Project

---

08/22/2006

The National Charter School Research Project (NCSRP) at the University of Washington has convened, since May 2005, a Consensus Panel of national experts on testing and evaluation to assess the evidence on charter school achievement. The Panel’s white paper, Key Issues in Studying Charter School Achievement: A Review and Suggestions for National Guidelines (May 2006) critiqued possible research methods and showed how more valid research could be done.

Two of the white paper’s conclusions can help readers interpret the report published today by the National Center for Education Statistics, "A Closer Look at Charter Schools Using Hierarchical Linear Modeling":

• First, “snapshot” studies, which compare different schools on a single test administered once, cannot be used to assess the effects of a student’s attending one kind of school versus another. With only one test score it is impossible to know whether outcomes are due to school quality or to something else, e.g., differences in students’ preschool learning or the quality of schools attended previously.
• Second, studies that lack good measures of student attributes can miss important differences in the student bodies of different schools. Student achievement is a complex result of student attributes, prior learning, and school effectiveness. Unless researchers can be sure that the students in two schools are exactly alike, they cannot confidently attribute performance differences to school effectiveness. The data available for the NCES study leave open the possibility that student populations that look alike are in fact very different.

Good researchers understand these limitations. That is why the authors of the NCES report make it clear that they know such a study can report outcomes but can’t say why they occur. However, less careful popularizers and interest groups often ignore these limitations, invalidly claiming that achievement results are due to school quality. A “snapshot” based study can be valuable in raising questions about cause and effect, but it can’t answer them, no matter how sophisticated its methods.

As the Panel's white paper shows, people who want to know whether students benefit from attending charter schools can get good answers only by using data and methods that are up to the task. Two strong methods are possible:

• Comparing individual students’ test scores before and after entering charter schools, in order to judge whether students’ learning rates were higher or lower in charter than in non-charter schools.
• Comparing the scores of students attending charter schools with those of students who applied to the same schools but did not get in because all the seats were taken.

These methods allow researchers to distinguish the effects of attending a charter school from other factors that can influence outcomes, e.g., student characteristics and prior educational experience. States and localities are building multi-year test-score databases, and lottery-based studies are possible in many localities. These will produce much more valid results than any snapshot study, no matter how carefully the latter is done.

Please note that this NCSRP message draws from the Consensus Panel’s general recommendations, but the Panel has not yet reviewed and rated the new NCES study.

The Charter School Achievement Consensus Panel Members include:

• Julian Betts, University of California, San Diego
• Dominic J. Brewer, University of Southern California
• Anthony Bryk, Stanford University
• Dan Goldhaber, University of Washington
• Laura Hamilton, The RAND Corporation
• Jeffrey R. Henig, Columbia University
• Paul T. Hill, University of Washington
• Susanna Loeb, Stanford University
• Patrick McEwan, Wellesley College