Fox, Meet Henhouse

Depending on who you talk to, Congress is either on the verge of finally passing the long-delayed new version of the federal Higher Education Act, any day now, really, or time has run out and it's all going to get pushed off into 2009 and beyond. Either way, it's worth pointing out that, based on the language currently being considered, Congress is on the verge of doing something very smart and then immediately undermining that smart thing by doing something that's very, um, not smart.

Here's the smart thing: the new law would require colleges and universities to report graduation rates "disaggregated by gender, by each major racial and ethnic subgroup, by recipients of a Federal Pell Grant, by recipients of a [federal student loan], and by recipients of neither a Federal Pell Grant nor a [federal student loan."

This is a commensense and extremely important provision. The federal government spends many billion dollars per year on aid programs for financially needy students, and state governments spend billions more. A whole host of programs, research areas, and controversies exist surrounding the issue of college access for low-income undergraduates. Yet there is shockingly little data about what happens to those students once they get to college. We collect graduation rates for basketball and football players, but not for students receiving financial aid. This clearly needs to change.

Unfortunately, the bill proceeds to add this qualifier: "if the number of students in such subgroup or with such status is sufficient to yield statistically reliable information and reporting will not yield personally identifiable information about an individual student. If such number is not sufficient for such purposes, then the institution shall note that the institution enrolled too few of such students to so disclose or report with confidence."

This seems reasonable, no? It wouldn't make sense to calculate a graduation rate for, say, one student. But here's the problem: the bill as written would allow individual institutions to determine the threshold for "statistically reliable." This is a terrible, terrible idea. It is the road to madness, because inevitably different institutions will choose to interpret the provisions in different, self-serving ways. This is a certainty. Congress made the same mistake when it wrote No Child Left Behind and allowed states to determine which statistical tests to apply in deciding how to hold schools accountable for student performance. Some decided to use a p < .05 test of statistical significance; others chose p < .01. Some decided that five students was the minimum number for reporting results; others decided it was 100 students. Worse, many states change their statistical standards every year.

It's highly likely that at most institutions, students on financial aid graduate at a lower rate than the student body as a whole. This can be embarrassing, further giving colleges reasons to use the most expansive exclusion rules imaginable. And since there are thousands of higher education institutions out there, the U.S. Department of Education won't even be able to do what it does with states under NCLB, which is require them to make their statistical standards public and submit them for approval.

If those goes through as written, here's what will happen: after a number of years, the data will finally be released to the public. Researchers will quickly notice that many institutions with signficant numbers of students on financial aid aren't reporting results as required. When asked, the institutions will say they applied a statistical test, but won't say what the test was. Then the colleges that did report results will argue that, due to the inter-institutional variance in reliability standards, it's unreasonable to compare Pell grant graduation rates at one institution to another, which of course is pretty much the whole point of having them in the first place.

There's a better way, which is the way graduation rates are currently reported: institutions submit all their data, confidentially, to the National Center for Education Statistics, and then NCES applies a uniform statistical standard to all institutions. That's the way it already works with the race and gender disaggregation, and it works perfectly well: small n sizes aren't reported and qualifiers are attached to rates that are statistically less reliable. If this new language goes through the whole thing will end in tears and we'll have to wait another ten years to find out which institutions are doing a good job helping low-income students earn degrees.