Report Touting NCLB Success Cannot Support Overstated Claims

Methodology is too weak to support finding that student achievement has increased since passage of NCLB

EAST LANSING, Mich.— A report released last month by the Center on Education Policy (CEP) is being used to argue that student achievement has increased since the implementation of the federal No Child Left Behind law. A review of this report finds that it suffers from important weaknesses and that the wording of numerous findings and key conclusions imply a much stronger connection between NCLB and increased achievement than can be substantiated by the data.

The report, Answering the Question That Matters Most: Has Student Achievement Increased Since No Child Left Behind was reviewed for the Think Twice think tank review project by John T.Yun, a professor at the University of California at Santa Barbara.

The CEP’s report has already received widespread attention from the news media, including front-page coverage in the Washington Post. The U.S. Secretary of Education immediately pointed to the report as confirming NCLB’s success. As reviewer Yun notes, the report “is likely to be cited often in the upcoming debate on the reauthorization of the No Child Left Behind Act (NCLB).”

While Yun credits the report with attempting “to carefully analyze the complex issue of test score improvement before and after the implementation of NCLB in 2002,” he describes how shortcomings in the data and analyses may have “resulted in a much more optimistic picture of the impact of the legislation than the data warrant.” Additionally, Yun says, while the report’s title may convey the impression that it seeks to examine the direct impact of NCLB on student achievement, the report itself acknowledges that an analysis accomplishing this goal may be impossible. The possible effects of NCLB cannot be disentangled from the possible effects of the large number of other state and local policies aimed at raising achievement during the same period of time.

The review from Professor Yun contends that the most useful and important finding in the new report is its explanation of current weaknesses in state data availability. The authors had great difficulty in obtaining and analyzing state-level achievement data that should be readily available; until this situation is improved, researchers such as those at CEP will be faced with many of the same obstacles encountered here. But Yun stresses that this finding was given far too little attention and should not have been overshadowed by the problematic analyses of student achievement.

Regarding the student achievement analyses, Yun does credit the report with offering thoughtful approaches and concludes that the report does represent progress toward more comprehensive examination of the outcomes of the law. He also commends the report for cautionary notes that help readers understand some of its limitations—yet not the three that he identifies as most serious:
The report’s look at whether achievement scores have increased since 2002 (its ‘trend analysis’) used an approach that had a likely unintended effect of analyzing a sub-sample of states that was biased toward those that were most likely to have inflated test scores.

The report’s finding of narrowing achievement gaps between groups of students suffered from the same weaknesses as its trend analysis, and also suffered from the problem of small sample sizes among some of the studied racial and ethnic groups, meaning that each of the percent-proficient estimates were effectively unreliable.

Selection bias is also likely to have “seriously damaged” the value of the report’s analysis of pre- and post-NCLB outcomes, which found yearly gains in test scores greater after NCLB took effect rather than before. In fact, Yun points out that the authors selected for “exactly the wrong group” of states to consider if the goal of the analysis was to examine the impact of NCLB, because the approach used had the effect of screening out those states that changed their approaches following NCLB’s passage.

These limitations seem particularly salient given the report’s prompt interjection into the public debate. U.S. Secretary of Education of Education Margaret Spellings, for instance, used the report to argue for NCLB reauthorization. Her official statement said, “This study confirms that [NCLB] has struck a chord of success with our nation’s schools and students. … We know the law is working, so now is the time to reauthorize [it].”

Commenting on this new review in light of such reaction the report, Professor Kevin Welner of the University of Colorado at Boulder, co-director of the Think Twice project, focused on the issue of publicity. “The methodological problems pointed out by Professor Yun are important and should be carefully considered by any policy maker or researcher who makes use of the study. But the bigger problem here seems to be in the packaging and subsequent publicity. Neither the data nor the analyses in the report are anywhere near strong enough to meaningfully support the report’s title, ‘Answering the question that matters most,’ nor can the report support the sort of puffery we see from Secretary Spellings.”

Find the complete review by John Yun as well as a link to the Center on Education Policy report at: http://www.greatlakescenter.org.