Value Added

Colleges and universities distinguish themselves from one another in lots of different ways-- scholarly reknown, the size of the endowment, success on the athletic fields, etc. But the most commonly-used measure is probably the "quality" of the freshman class, as measured by standardized tests like the SAT and ACT. Average incoming SAT scores at University of Texas campuses, for example, look like this:


















The Austin and Dallas campuses are getting students at 1200 and above while the non-selective regional campuses like Pan American and Permian Basin are below 1000. This conforms with nearly any measure of prestige and status one could name: Austin is an internationally known, Research I, AAU institution with a multi-billion dollar endowment and a football team that was lucky enough to beat my Ohio State Buckeyes in the Fiesta Bowl last month, not that I'm bitter. (Although: "Colt McCoy"? Really?) Permian Basin has none of these things, and probably never will.

But SAT scores leave the question of college student learning unanswered. It's odd, the way we give colleges credit for how their student did on a test they took while they were juniors in high school. Colleges argue that high SAT scores are an implicit quality signal because they reflect high demand, but the demand may just be for the prestige and the football team and the nice facilities and the chance to hang around with other students who also have high SAT scores. To really get a handle on learning, it makes more sense to test a sample of freshmen and a sample of seniors, and see how they compare. And in fact the University of Texas system has done exactly that, using the Collegiate Learning Assessment. Here's what they found:





















Each block on the graph shows two data points: freshman and senior scores on the CLA. As you'd expect, freshman scores correspond fairly closely with SAT scores: Austin and Dallas have the highest, regionals like Permian Basin the lowest, and the rest are in between. Much more interesting is growth. While Austin students arrive at high levels, they don't seem to improve very much while they're in college--the difference of 53 points is less than half the national average of 111 points. This may because of some sort of "ceiling effect," or it may be that elite universities don't focus much on improving students who arrive in great shape to begin with. Pan American and Permian Basin have very similar freshman scores, but Permian Basin's growth is more the double that of UTPA -- 197 to 90, bringing students from well below the national average on entry to above it on completion.

The CLA, it should be said, is not the be-all and end-all of college assessment. It's a general assessment of analytic reasoning, critical thinking and communications skills that doesn't measure mastery of the disciplines. It's subject to measurement and sampling error, like any standardized test. But it's also being used at hundreds of institutions and is based on a lot of smart thinking in psychometrics. It should be the beginning of much more attention to how much students learn while they're in college. This is how we ought to be thinking about success and prestige in higher education.

The CLA results also highlight severe limiations in the way we credential college students, and the vast differences in ability among students who are all pushed through a system that in many ways assumes they're the same. Note that despite the unusual growth at Permian Basin, seniors there still score well below freshmen at Austin. The premium given in the job market to degrees from highly selective institutions is, in that sense, quite rational; students could literally learn nothing while at an elite college and still outperform most other college grads.

The real inefficiencies and failures in the labor market occur at the individual level, particularly among the great masses of students with degrees from non-selective and thus largely undifferentiated instutitions. Lets say you're a very bright student who, for financial or family reasons, chooses to attend college at a local four-year institution like UTPB, except in a state that doesn't publish value-added measures like the CLA. You work hard, excel at your studies, and graduate at the top of your class. Do you get credit for this? No, you do not. The market cares little about college grades because they're opaque and inconsistent. So it assigns you the average value of a UTPB freshman, based on SAT scores, because that's the only comparable information it has. By the same token, the guy who finished last at Austin is over-valued in the market. And of course the brilliant person who never got a college degree at all is left completely out in the cold.

Sample-based measures like the CLA are only the beginning; what we really need to do is start attaching a lot more useful information to individual college credentials while also making the credentialling process itself more open and flexibile, less about having been taught by some kind of formal institution and more about having actually learned something real.