Partially Flawed Research

Report on High Achievers Offers Useful Analyses but Overstates Policy Implications

EAST LANSING, Mi-A recent report suggests that high-achieving students are losing out under the No Child Left Behind Act and recommends incentives for schools to better serve such students. A Think Twice review of the report praises its focus on high achievers, especially those from resource-poor schools, but concludes that the report’s presentations of findings and policy implications inappropriately overreach.

The two-part report, High-Achieving Students in the Era of NCLB was published by the Thomas B. Fordham Institute and was reviewed by Professor Gregory Camilli of Rutgers University. The report comes as Congress considers a redesign of NCLB, and Camilli notes it is framed to influence that discussion.

The report consists of 1) an analysis of mathematics and reading achievement among higher- and lower-achieving students as measured by a national standardized test, and 2) the results of a survey of teachers about how schools serve high achievers while meeting NCLB’s requirements.

Analyzing results of the National Assessment of Educational Progress (NAEP) in reading and mathematics for students at the 10th (lower-achieving) and 90th (higher-achieving) percentiles on those exams, the report concludes that the gap between the two groups has narrowed since NCLB took effect. This result was due to achievement gains by low rather than higher achievers. The report then suggests that lower-income and minority high achievers may be at risk for losing out on opportunities under NCLB.

The second part of the report highlights findings concerning teachers’ perceptions and beliefs regarding the education of higher achievers. For instance, 86% of the teachers responding indicated that it is more important to focus on achievement among all students, regardless of their advantages, rather than to focus primarily on struggling, disadvantaged students.

Camilli applauds the spotlight placed by the report on Black, Hispanic, and poor high achievers, and he concludes that it provides a “meaningful statistical description of these student populations.” But he also offers several criticisms of the report:

*While the report includes acknowledgements that its correlational analyses cannot be used to draw causal inferences, such inferences nonetheless are used pervasively to bolster policy recommendations. Camilli is particularly critical of such over-reaching in the report’s foreword, written by Fordham’s Chester Finn and Michael Petrelli, as well as in the executive summary.
There are in fact two NAEP data sets from which gap statistics can be calculated. The report’s analysis of the gap between high and low achievers uses only one. Camilli shows that with the other data set, much less convincing results are obtained, thus limiting the generalizability of the gap trends presented in the report.

*The report suggests policies that would increase stratification of educational opportunities, contradicting “a large body of research” that questions the efficacy of tracking students by perceived ability. This research is not discussed in the report.

*Response bias to the survey of teachers may be a problem. It received only a 15% return rate and did not provide appropriate breakdowns by such characteristics as grade level and urbanicity that might have better informed readers. The wording of certain questions may have inadvertently led to response bias.

*Camilli concludes that the report’s two studies inappropriately “attempt to inform broad policies on the strength of two fairly narrow analyses.”

Find Gregory Camilli’s review and the The Fordham Institute’s report at: http://www.greatlakescenter.org/.