Margins of Error

Kevin Drum wrote a good post a couple of weeks ago about statistical illiteracy in the media, viz. the widespread tendency to characterize election poll results in which one candidate's percentage point lead is equal to or less than the poll's statistical margin of error (MOE) as a "statistical tie" or "dead heat." Kevin notes:

...probability isn't a cutoff, it's a continuum: the bigger the lead, the more likely that someone is ahead and that the result isn't just a polling fluke. So instead of lazily reporting any result within the MOE as a "tie," which is statistically wrong anyway, it would be more informative to just go ahead and tell us how probable it is that a candidate is really ahead. Here's a table that gives you the answer to within a point or two:

As Kevin notes, if Obama is up by three points, and the MOE is three points, it's 84% likely that he'll be the next President of the United States. That's very different than 50% likely, i.e. an actual tie.

This is directly relevant to education because most states use precisely the same statistical techniques when deciding whether a school has made Adequate Yearly Progress (AYP) under No Child Left Behind. If, say, 65% of students need to pass the test in order to make AYP, and only 62% pass, but the state determines an MOE of 4 percentage points, then the school makes AYP because the score was "within the margin of error."

This is silly for two reasons. First, unlike opinion polls, NCLB doesn't test a sample of students. It tests all students. The only way states can even justify using MOEs in the first place is with the strange assertion that the entire population of a school is a sample, of some larger universe of imaginary children who could have taken the test, theoretically. In other words, the message to parents is "Yes, it is true that your children didn't learn very much this year, but we're pretty sure, statistically speaking, that had we instead been teaching another group of children who do not actually exist, they'd have done fine. So there's nothing to worry about."

Second, per Kevin's chart above, the idea that scores that fall below the cutoff but within the margin of error are statistically indistinguishable from actual passing scores is incorrect. This is particularly true given that, while opinion polls almost always use a 95% confidence interval to establish their MOEs, most states use a 99% confidence interval for NCLB, which results in substantially larger margins of error around the passing score. But states do it anyway, because many of them basically see NCLB accountability as a malevolent force emanating from Washington, DC from which schools need to be shielded by any means necessary.

Think of it this way: let's say your child is sick and you bring him to the doctor. After the diagnosis is complete, you and the doctor have the following conversation:

Doctor: My diagnosis is that your son has pneumonia and needs to be hospitalized.

You: That's terrible! Are you sure?

Doctor: Well, there are few absolute certainties in medicine. It's possible that he only has bronchitis. But I'm pretty sure it's pneumonia.

You: How sure?

Doctor: 84% sure.

What would you do? Would you (A) Check your son into the hospital? Or would you (B) Say "Hey, there's a 16 percent chance this whole thing will work itself out with bedrest and chicken soup. Let's go that way."

States implementing NCLB nearly always choose option (B). That's because they see the law as a process for making the lives of educators worse, not what it actually is: a process for making the lives of students better.