False Positives, Seed Corn, Etc.

This story in today's Post about the 27 schools in DCPS that are in "restructuring" mode under NCLB is well worth reading. Faced with a federal mandate to choose among various reforms--charter school conversion, private management, replacing teachers and principals, etc.--Chancellor Rhee decided to send groups of outside teachers, parents, students and educators into the schools in order to gather some qualitative data beyond what the test scores say. Their conclusion: many of these schools are very bad.

This illustrates an important point about correlation. NCLB is often criticized for relying almost exclusively on standardized test scores in reading and math. Beyond the fact that this ignores many other important subjects and broader educational goals, the tests themselves can only measure certain domains within reading and math, with imperfect accuracy. The efficacy of NCLB, therefore, is heavily dependent on the correlation between school-level test scores and school-level performance on everything that tests don't measure. If the correlation is high--if schools with terrible test scores also tend to be terrible in many other ways--than the system works. If the correlation is low, it doesn't.

This article suggests that, in DC anyway, the correlation is pretty high. I'm not surprised. The "tests don't measure everything" argument has always struck me as semi-convincing in terms of false negatives--one could plausibly imagine a school using some kind of maniacal eat-your-seed-corn test prep strategy to make AYP, even though it's actually not any good. It's a futile strategy in the long term, but it might work for a few years.

It seems very unlikely, by contrast, that non-correlation is going to produce a lot of false positives. In other words, if you miss AYP for six consecutive years--that's the criteria for ending up in restructuring, like the 27 DC schools--there's very likely a legitimate reason involving poor educational practices that need to be fixed. And for the most part, that's what the teams found. I realize this wasn't a formal study and so the observers may have gone in looking for problems. But it's hard to get past comments like these, from the students:

The question to a focus group of Dunbar High students was: What did they like best about going to school there? "Freedom," said one who takes Advanced Placement classes at the school in Northwest Washington. "We can do whatever we want at this school. That's the only good thing about this place."

The report notes that "In many cases, [the reviews] depict rudderless and cheerless institutions where students wander the halls with impunity during class and staff members have all but given up trying to maintain order." It also says:

But the theme resonating most powerfully in the reports is student frustration with the lack of academic rigor. Although there are always a few inspiring instructors, students -- none of whom were named -- said too many teachers approached their jobs with indifference and low expectations. "Teachers don't teach us a thing throughout the entire period," said one Lincoln student. "When visitors come, they start working." At Anacostia High in Southeast, evaluators described a history class exercise where students were prompted to respond to the question, "Where is your favorite place to shop?" None of the randomly selected students at Dunbar High responded positively when asked whether the school was preparing them for college. Pressed further, they said they didn't even feel ready for the workforce beyond high school.

Here's the one phrase I'd encourage you to keep in mind: "lack of academic rigor." Lack. There's a common way of thinking about school reform in urban areas like DC that goes something like this: These students come from tough home lives and have many barriers to learning, and as a result they're not doing very well academically under the standards we have. Increasing those standards and raising expectations is, therefore, punitive and counterproductive, a case of making a hard life harder still, and will result in more dropouts and further disengagement from education.

But when you talk to actual students--and this is by no means an isolated incident--they tend to make the opposite point, which is that low expectations are the problem. Lack of rigor isn't just symptomatic of bad teaching, it is bad teaching. The state of adolescence is such that you know enough to know that you should be held to high standards, but you also need adult supervision to enforce those standards. If NCLB is identifying and fundamentally changing schools where students get nothing but the freedom to do whatever they want, it must be doing something right.