Those tricky charts

The Fordham Foundation's report comparing high and low achievers under No Child Left Behind had me initially convinced. I wrote an entire blog post claiming that it showed the merits of NCLB, that the law could be credited for closing the achievement gap, and pushing for a future system of accountability that held schools responsible for the education of all children from all backgrounds with all abilities.

In my post I reiterated the gains published by Fordham. They say:

Low-achieving students made solid progress on the National Assessment of EducationalProgress (NAEP) from 2000 to 2007 (an accomplishment surely worth celebrating, even though these students are still far, far behind). Meanwhile, however, the progress of our top students has been modest at best.

Their charts showed me graphically what I had just read. While the chart above purported to present how low achieving students increased in the years following the enactment of the law, the one at left supposedly showed no major closing of the achievement gap before NCLB.

But these aren't the charts we should have seen. I took these same two Fordham charts and combined them below. The solid line represents where Fordham drew the cut-off for NCLB, the year 2000. But No Child wasn't signed into law until January 2002. The first NAEP tests measuring its true impacts could not have been until 2003, represented by the dotted line.

When we make this correction, the claims in the report do not seem to stand up as well. The lower tenth of performers made gains throughout the chart, but especially from 2000 to 2003, where they gained 13 points to their high achieving peers' six. Notably, this accounts for almost all the gain claimed in the Fordham report.
















Fordham's argument, that we've focused too heavily on equity issues while neglecting excellence, could have been made without the misdirection. Along with this analysis of NAEP data, the report also featured a teacher survey. It supported their earlier conclusions, and could have stood on its own merits. Instead, they published both together, and the data doesn't show exactly what they claim. All we're left with, in the end, is debating tough choices.

Update: Mike Petrilli responds, citing the difficulties in defining the "era of NCLB." This is indeed a tough task. It's even tougher to put statements like these:

The fairest approach is to point out the large gains in NAEP scores in the period around 1998-2003 and acknowledge that NCLB’s association with these gains is unknown.

into charts.

My broader point is that if you compare scores across a longer spectrum, from 1992 in reading and 1990 in math (as Fordham did), the lowest ten percent of achievers narrowed the achievement gap from their top ten percent peers by 9, 2, 1, and 4 points, respectively, for 4th and 8th grade math and 4th and 8th grade reading. Those changes are in scale scores, not percentages. That's not exactly a sign of crisis.