Teaching, the Null Hypothesis, and the Status Quo

I've had a couple of off-line conversations in the last week--one about measuring teacher effectiveness, the other about college graduation rates--that both led me to try answer the eternal question of : Why are academics so often wrong about public policy questions?

The short answer is: they're trying to answer the wrong question.

The somewhat longer answer is this: Academics and researchers are trained to think about evidence in a specific way. Their default position is the null hypothesis: unless you can prove something is true, it's not true. This is a completely appropriate way to approach the kind of work that academics do. If your job is to add bricks to the edifice of collective human knowledge, you want to make sure they can stand some weight--otherwise, the whole thing can come crashing down. The generally accepted standard for "statistical significance," for example, is 95% confidence, which means that at least 19 times out of 20, the relationship you're observing is real and not the result of random variation. Nobody disputes this standard, and indeed people sometimes hold out for 99% confidence or more.

The essential public policy question, by contrast, is not: "Is the null hypothesis true?" It's: "Should we keep doing what we're doing, or do something else?" It's a choice between change and the status quo. Neither of those alternatives deserves any special consideration; we should (allowing for the transition costs of change) choose whichever is most likely to achieve whatever policy goals we may have. In other words, the standard in public policy isn't 95%, it's whatever is most likely to be best: 51%. Of course, something closer to 95% would be better, but policy choices are rarely that obvious.

Crucially, in the policy world, choices cannot be delayed or avoided, because not changing is, itself, a choice. A vote against change is a vote for the status quo. Take public education. There are 50 million students in public school today in this country. They're going to be there again on Monday morning, and on Tuesday, and on Wednesday, and in the days and weeks after that. Their schools will likely remain as they are unless we change them. Not changing them endorses that sameness. And I think most reasonable people agree that for too many students, the schools aren't working well enough.

Yet academics consistently treat policy questions like academic questions. They mistake the status quo for the null hypothesis. For example, one alleged social scientist recently concluded that, given some unresolved questions about a proposed value-added teacher effectiveness method, "it's not ready." From her perspective, the question is: can we be really, really sure--say, 95% sure--that value-added measures are accurate?

If we had infinite time and resources to construct the perfect teacher evaluation process, this might be the right question. But of course, we don't. Instead, we have schools--which will, I must emphasize, re-open their doors in less than 72 hours, whether we resolve these issues over the weekend or not--where the status quo process for evaluating teachers is perfunctory, inaccurate, and all but useless. It is a process that allows very bad teachers to stay in their jobs (If you don't believe me, read this). In that context, "it's not ready" is exactly the same as saying "let's keep the current terrible system," because that's the policy choice currently on the table, today.

In this way, the academic approach to public policy, where all changes must meet academic standards of proof, is biased toward the status quo in a huge and damaging way. We're sticking with policies that everyone knows are bad because some people aren't quite sure enough that changes would be good.

Draining the Pool

The news that Lynn Olson, Education Week's senior correspondent, is decamping to the Gates Foundation after more than two decades of writing thoughful trend pieces and news analyses reflects a disquiting trend in American education: the number of experienced journalists writing about schools and colleges for national newspapers and magazines is reaching a disturbingly low level.


Peg Tyre, who has covered the beat for seven years at Newsweek is leaving the magazine at the end of the month under a buy-out program that's going to leave the newsmagazine with over 100 fewer staffers. Claudia Wallis, who has written many of Time's education covers, has left the magazine. Ben Wildavsky departed U.S. News a few years ago for the Kauffman Foundation. And The New York Times has reportedly spiked its regular Wednesday education coverage.


Such cuts are part and parcel of the financial woes inflicted on print media by the advent of Internet advertising. But the collateral damage to education journalism is substantial. There are today very few journalists with the knowledge and experience to write authoritatively for national, non-specialist audiences. There's been a proliferation of education bloggers ready to share their opinions (yes, I'm writing this on a blog). But smart, analytic long-form writing on education's big themes, the sort of work that Lynn did for a long time from her independent perch at Education Week, is becoming harder and harder to find. As Eric Alterman wrote recently in The New Yorker about journalism generally in the Internet era: "We are about to enter a fractured, chaotic world of news, characterized by superior community conversation [via blogging], but a decidedly diminished level of first-rate journalism."


The news industry's economic woes eventually may sort themselves out. Until then, we need to find new ways to support the production of first-rate writing about education in national general-interest publications. Several foundations have taken steps in that direction by funding a new "public editor" position at the Education Writers Association and education fellowships at the Columbia Graduate School of Journalism. But we need to do much more if want to elevate education to the status it deserves in the national public policy conversation.

Finn Speaks

Proving once again (as if it were even necessary at this point) that when The Quick and the ED speaks, people listen, Fordham's Checker Finn dips his toe in the blogosphere for the first time, by calling foul on the "bizzare piggybacking and ahistoricism" of an upcoming Heritage Foundation event titled "25 Years After A Nation at Risk: Returning to President Reagan's Vision for American Education." The event description says:

When [A Nation At Risk] was released in 1983, President Reagan outlined a bold vision for reforming education. He called for increasing parental choice, limiting federal government involvement, and restoring state and local control in education. But conventional wisdom and education reforms have followed a different path over the past quarter-century – increasing federal authority and expanding government control of education.

As Finn notes, Reagan may have said some stuff about vouchers and whatnot when the report was originally released, but he quickly realized that A Nation at Risk advocated for a completely different agenda--an agenda that he then embraced, an agenda that in fact tracks fairly closely with the past quarter-century of education reforms that Heritage derides.

This is a good illustration of the deep philosophical division among right-of-center folks when it comes to the public schools. On the one hand, you've got the Heritage / Cato types who basically see public education as a gigantic, unreformable black hole exerting immense gravitational pull on the public treasury, a prime generator of demand for the taxation they hate above all else and a revenue source for the unions that are a close second. Thus: vouchers, privatization, abolishing the U.S. Department of Education, whatever.

Then there's the Fordham / Checker Finn / A Nation at Risk perspective, which also starts from a very critical view of the present public education system, but concludes that the answer lies in more rigorous standards and greater governmental accountability for results. There are elements of the basic libertarian / authoritarian divide here; Cato trusts in the magic of the market and parental choice, while Finn thinks the answer lies in more rigor, seriousness, professionalism, and tough accountability.

One of the major findings of A Nation at Risk, for example, was that high school students were taking a mish-mash of low-level courses that didn't prepare them for college or anything else. The report called for students to take a "new basics" curriculum -- four years of English, three years of math, three years of science, etc. The libertarian would leave this up to local schools and parents to decide, while others would say no, everyone needs to learn these things whether they like it or not, and it's the responsibility of society and schools to enforce these standards.

While there are some commonalities between the Heritage and Fordham approaches to education--suspicion of unions, good feelings toward choice-based reforms, and a general sense that the schools waste vaste amounts of money--there are also areas that are fundamentally irreconcilable, and it's silly to pretend otherwise.

Hearing Back on Benwood

I've been getting a lot of feedback on the report about the Benwood Initiative that we recently released. Some who appreciated the "nuance" of the findings, others who had great methodological questions, and a smattering of folks who offered terse commentary that can be summed up as "you're saying it takes everything to change the culture of schools and raise student performance, which in turn says nothing and makes it impossible to make choices and so basically there's nothing we can do." Well, I don't know about everything but yes, you do have to change the culture of schools and that does take a whole lot more than any single policy change. And I would hope that could help inform the choices that schools and districts make--so they don't put all their eggs into one basket and expect immediate hatching. I've heard from a few district and school leaders too, and have engaged in some back-and-forth with them. This I find the most heartening-- to hear their thoughtful comments and their ideas about how this relates to their own schools and districts.

The findings have also been misread and misreported. The Baltimore Sun ran a story about Baltimore County teachers who may have to reapply for their jobs as part of restructuring. In it, they reported, "Education Sector, has a new report studying inner-city schools in Chattanooga, Tenn., that made dramatic gains. While those schools replaced some staff members, the report found, the teachers who were most successful were veterans who went through extensive professional development."

To clarify, our research found that about 2/3 of teachers who were teaching in Benwood schools were rehired during the reapplication process. These were not necessarily veteran teachers nor is there any evidence to suggest that veteran teachers who were rehired were more successful than newer non-veteran teachers. What the report shows is that a group of mostly the same teachers improved over time, debunking the notion that the slate was wiped clean of existing teachers and replaced with new and better ones.

Flypaper Cometh

The Thomas B. Fordham Institute has launched a new education blog, Flypaper. This is a welcome addition to the edublogosphere and one that I imagine will quickly become a staple of most people's daily edublog shortlist. Why I don't always agree with the folks at Fordham, they're smart and have a lot of interesting, often provocative ideas about education, generally from a kind of reformist center-right perspective. I note, however, there have been no posts from Fordham head honcho Checker Finn as of yet. Finn may not want to admit this to himself, but he was born to blog. Give in to the inevitable, Checker!

It Must Be Mine...Oh Yes...

I was at a concert at the 9:30 Club tonight and saw some guy wearing this T-shirt as I was walking out. Now, as you all know, we work pretty hard here to bring you the best education policy analysis going, and this is part of our job. But the TV recaps? The travelogues and concert reviews? That stuff pretty much happens on my own time. So this seems like a good opportunity for some enterprising readers to band together and do me a solid in return by buying me one of these T-shirts. I'm a size Large, send it to me c/o

Education Sector

1201 Connecticut Ave NW, Suite 850

Washington, DC 20036

Shafting Poor Students in Higher Education

There's been a fair amount of analysis in recent years about the various ways that low-income students are getting short-changed by higher education financial aid systems, with both states and individual institutions devoting a larger percentage of financial aid dollars to so-called "merit aid" programs that disproportionately benefit well-off students, i.e. those who need the least help. Crucially--and I can't emphasize this enough--much of this aid is not based on merit but rather amounts to colleges throwing $5,000 to $10,000 towards a rich student in the hopes that he/she will be flattered into enrolling and his/her parents will write a check for the remaining thirty-five grand, plus additional donations to the alumni fund down the road.

But there's been surprisingly little attention given to an arguably bigger and more important funding inequity: the public colleges and universities that lower-income, lower-achieving students tend to attend receive and spend a lot less money than the public universities where the wealtheir higher-achieving students go to school. In some states, the financial disparity, even after excluding spending on research, can run close to $10,000 per student or more. In K-12 education, number like that frequently get states thrown into court where they face, and lose, huge school funding lawsuits resulting in billion dollar settlements. In higher education, we think it's justice. This is the subject of my column in today's InsideHigherEd.

Donor Quirks

Yesterday's New York Times article about the strings some donors attach to their gifts to universities reminded me of my own alma mater. Thanks to two dog-loving donors, William and Mary houses the second largest collection of cynogetica: books about dogs. From the Swem Library website, "This is the second largest collection of books about dogs in this country and continues to grow through its own endowment. It contains scholarly work that dates back to the sixteenth century as well as children's literature, breed guides, and the records of the American Kennel Club."

Now there's a good use of an endowment. Hey, it's the reason I went there.