Teaching, the Null Hypothesis, and the Status Quo

I've had a couple of off-line conversations in the last week--one about measuring teacher effectiveness, the other about college graduation rates--that both led me to try answer the eternal question of : Why are academics so often wrong about public policy questions?

The short answer is: they're trying to answer the wrong question.

The somewhat longer answer is this: Academics and researchers are trained to think about evidence in a specific way. Their default position is the null hypothesis: unless you can prove something is true, it's not true. This is a completely appropriate way to approach the kind of work that academics do. If your job is to add bricks to the edifice of collective human knowledge, you want to make sure they can stand some weight--otherwise, the whole thing can come crashing down. The generally accepted standard for "statistical significance," for example, is 95% confidence, which means that at least 19 times out of 20, the relationship you're observing is real and not the result of random variation. Nobody disputes this standard, and indeed people sometimes hold out for 99% confidence or more.

The essential public policy question, by contrast, is not: "Is the null hypothesis true?" It's: "Should we keep doing what we're doing, or do something else?" It's a choice between change and the status quo. Neither of those alternatives deserves any special consideration; we should (allowing for the transition costs of change) choose whichever is most likely to achieve whatever policy goals we may have. In other words, the standard in public policy isn't 95%, it's whatever is most likely to be best: 51%. Of course, something closer to 95% would be better, but policy choices are rarely that obvious.

Crucially, in the policy world, choices cannot be delayed or avoided, because not changing is, itself, a choice. A vote against change is a vote for the status quo. Take public education. There are 50 million students in public school today in this country. They're going to be there again on Monday morning, and on Tuesday, and on Wednesday, and in the days and weeks after that. Their schools will likely remain as they are unless we change them. Not changing them endorses that sameness. And I think most reasonable people agree that for too many students, the schools aren't working well enough.

Yet academics consistently treat policy questions like academic questions. They mistake the status quo for the null hypothesis. For example, one alleged social scientist recently concluded that, given some unresolved questions about a proposed value-added teacher effectiveness method, "it's not ready." From her perspective, the question is: can we be really, really sure--say, 95% sure--that value-added measures are accurate?

If we had infinite time and resources to construct the perfect teacher evaluation process, this might be the right question. But of course, we don't. Instead, we have schools--which will, I must emphasize, re-open their doors in less than 72 hours, whether we resolve these issues over the weekend or not--where the status quo process for evaluating teachers is perfunctory, inaccurate, and all but useless. It is a process that allows very bad teachers to stay in their jobs (If you don't believe me, read this). In that context, "it's not ready" is exactly the same as saying "let's keep the current terrible system," because that's the policy choice currently on the table, today.

In this way, the academic approach to public policy, where all changes must meet academic standards of proof, is biased toward the status quo in a huge and damaging way. We're sticking with policies that everyone knows are bad because some people aren't quite sure enough that changes would be good.