The Burden of Proof

Paul Basken (one of the best higher education reporters in the business IMHO) filed a short piece($) in the Chronicle a few weeks ago about struggles to improve the quality of teaching in engineering. He wrote:

After a close-up look at 40 American engineering schools, the Carnegie Foundation for the Advancement of Teaching has released a new report on the matter, but the diagnosis is old news: A widespread emphasis on textbook-heavy theory over hands-on practice discourages many students and leaves the ones that remain unprepared for real-world problems. With the difficulty long known, why have solutions been so elusive? Among the reasons cited by college leaders: a faculty culture resistant to change, and perceived pressure from accreditors.

And:

The "problem-based approach" in Georgia Tech's biomedical-engineering program includes asking sophomores to spend an entire semester exploring a big-picture question, such as how to keep the blood supply safe from the AIDS virus, said Laurence J. Jacobs, a professor of civil and mechanical engineering and an associate dean at Georgia Tech. Other colleges are having a much more difficult time introducing such changes in their traditional engineering programs because of faculty members who "are very, very protective of their curricula," Mr. Jacobs said. Changing faculty attitudes is the key, said an author of the Carnegie report, Sheri D. Sheppard, a professor of mechanical engineering and an associate vice provost for graduate education at Stanford University. The science foundation has spent millions of dollars trying to encourage universities to break up old styles of teaching, and it still couldn't overcome the "cultural issue of change" among faculty members, Ms. Sheppard said.

I went to college, four years of undergrad, two more in grad school. Both of my parents are retired college professors, and I recently taught a graduate course in education policy for Johns Hopkins. But I'm not of higher education. I've spent my career in public policy, first in the executive and legislative branches at the state level, then in various non-profit think tanks in DC. And I am just baffled when I read things like this. Maddened. Because it seems to me that one could easily summarize the two paragraphs above as follows: 

People have known for a long time that college students learn more when they're actively engaged in learning via hand-on practice and other means. But many professors refuse to adopt these methods, because they don't want to and they don't have to.

Am I missing something? To be clear, I'm not advocating for some kind monolithic scripted curriculum. When I put my class together, I made choices about subject matter and methods that suited my expertise and instructional strengths and weaknesses. But it seems to me that the more autonomy faculty are given in the classroom, the greater the burden of proof to demonstrate that their choices are actually working, with that proof being based, in significant part, on some evidence of what students learn. Isn't that what higher eduction is all about--evidence? And if the methods or approaches aren't working, they shouldn't be allowed to continue, period, regardless of who the instructor might be. Blaming this problem on "culture" is a dodge, a way of obscuring responsibility, as if faculty are helpless victims of some larger infectious mindset and not professionals who are, as such, responsible for the choices they make. 

Invest in the Future

This morning's New York Times has a good read analyzing what we can learn from Japan's Lost Decade, a period where they suffered a real estate bubble, pumped government spending as a stimulus, but could not raise economic growth. Here's the takeaway:

Japan’s experience also seems to argue for spending heavily to promote social development. A 1998 report by the Japan Institute for Local Government, a nonprofit policy research group, found that every 1 trillion yen, or about $11.2 billion, spent on social services like care for the elderly and monthly pension payments added 1.64 trillion yen in growth. Financing for schools and education delivered an even bigger boost of 1.74 trillion yen, the report found.

But every 1 trillion yen spent on infrastructure projects in the 1990s increased Japan’s gross domestic product, a measure of its overall economic size, by only 1.37 trillion yen, mainly by creating jobs and other improvements like reducing travel times.

Economists said the finding suggested that while infrastructure spending may yield strong results for developing nations, creating jobs in higher-paying knowledge-based services like health care and education can bring larger benefits to advanced economies like Japan, with its aging population.

In other words, a stimulus should not just be tax cuts or paying people to dig holes and then fill them in. A stimulus should invest in education. Let's hope our Senators are reading the same article.

Big Edu-cuts Proposed for Stimulus Package, Maybe

The latest tin can to come flying out of the maelstrom over the Congressional stimulus package is a document purporting to be recommendations for education cuts by the staff of centrist senators Ben Nelson, a Nebraska Democrat, and Republican Susan Collins of Maine, increasingly key players in the stimulus debate.

Passed along by folks with a big stake in the game, the document suggests that staff want to reduce education spending dramatically as part of cuts totaling $78 billion to the sprawling, $900 billion Senate version of the American Recovery and Reinvestment Act.

The $15 billion "state incentive grants" that the Obama administration wants to use to leverage school reform would be eliminated under the purported Nelson\Collins staff plan. The plan would cut $25 billion out of the $76 billion in "state stabilization" money intended to stave off teachers lay offs. And it would cut in half proposed increases from special education, Title I monies for disadvantaged students, Head Start, and teacher-quality partnership grants. Nearly $14 billion in new money for college Pell Grants would stay. Stay tuned.

Bill-Board

Bill Gates, borrowing a tradition from his mentor Warren Buffett, has published his first Annual Letter, his reflections on the work of the Bill and Melinda Gates Foundation, where he has been full-time since leaving Microsoft last summer.

Education commentators have focused on what they’ve characterized as a quasi-mea culpa in Gates’ discussion of the foundation’s 9-year, $2 billion investment in high school reform, particularly its effort to transform the nation’s large, often-alienating comprehensive high schools into smaller, more-personal educational environments. “We are trying to raise college-ready graduation rates, and in most cases, we fell short,” Gates writes, a declaration that some have interpreted as a suggestion that abandoning comprehensive high schools for smaller alternatives is an unfruitful reform.

That’s an unfortunate interpretation, because anyone who has spent time in the nation’s urban (and, for that matter, many suburban) high schools knows that the anonymity that pervades such schools contributes to a debilitating culture of apathy and alienation and is a root cause of the academic failure that afflicts so many American public secondary schools.

Not all large high schools are bad, of course. And not all small ones are successful--as the Gates Foundation discovered the hard way. But small schools are more likely to create the sense of connectedness among students and teachers, the sense of being known and valued, that motivates people to work hard. They encourage stronger bonds between students and teachers and generate a level of genuine caring and mutual obligation between them that is found far less frequently in large, comprehensive high schools. Small schools, in other words, are more likely to produce the conditions that make learning possible.

As Bill Gates points out in his Letter, that’s not enough. You also have to have high standards, a sound curriculum, and good teachers. Smaller schools are a means to an end, not an end in itself. But studies involving rural, suburban, and urban high schools have found that student and teacher attendance are typically higher in small schools. So are student involvement in extracurricular activities and graduation rates. Teacher turnover and student disciplinary problems are lower.

Because these results are necessary, but not sufficient, Gates has shifted much of its high school reform funding to networks of charter schools like KIPP, High Tech High, and Aspire Public Schools. As charter schools, they have more autonomy to address the other side of the reform equation—rigorous academic expectations, attracting top leaders and teachers, revamping the school day and school year to maximize instructional time, etc.

The Gates Foundation also gravitated to charter school networks because it learned that it’s a lot easier to establish a culture of success in a new school than it is to try to graft such a culture onto existing schools. Yet the schools’ powerful cultures, their sense of purposefulness, is also very much derived, their leaders are quick to point out, from the schools’ small size. The average charter school enrolls under 300 students, less than half the enrollment of traditional public schools, while some urban comprehensive high schools have as many as 5,000 students.

Unfortunately, there are only a relative handful of the new, high-performing charter schools that Gates praises in his letter, at most a couple of hundred that have made significant progress in closing achievement gaps for disadvantaged students. The financial and other challenges of increasing the number of such schools significantly are daunting. Which, perhaps, leads the Gates Foundation and others committed to school reform back to the traditional public school system—bearing the lessons it has learned from KIPP and other school networks.

Footnote: As some people know, I wrote a Gates-funded book several years ago called High Schools on a Human Scale, in which I made a case for abandoning comprehensive high schools, an argument that I had discussed more briefly in a 1991 book called In the Name of Excellence.

Good Point

Marguerite Roza of the Center on Reinventing Public Education observes that if the great recession forces school districts to cut their personnel budgets under "last in, first out" rules, they'll end up firing substantially more teachers than they otherwise would, because the last in tend to be younger and lower-paid and thus you have to fire more of them to save the requisite amount of money. She estimates that a 10 percent reduction in school personnel expenditures nationwide would result in 262,000 more lost jobs (and thus loss of health insurance, etc.) under seniority-based firing policies than under seniority-neutral policies. 

New Rule on Spending by States Lacks Teeth

That's the headline for an upcoming Chronicle of Higher Education article looking at whether a maintenance of effort provision included in last fall's Higher Education Opportunity Act will actually affect any states. The provision tied a small grant program to states maintaining at least five-year spending averages. Only one state, Rhode Island, is failing to do so. Over a period when enrollment in the state increased nearly ten percent, funding is now 5.2 percent below the five-year average, in unadjusted dollars. They'll lose $330,000 in federal funds, or .18 percent of their budget. Only two other states were within ten percent of the five-year average, Michigan at 3.9 and South Carolina at 6.1.

Value Added

Colleges and universities distinguish themselves from one another in lots of different ways-- scholarly reknown, the size of the endowment, success on the athletic fields, etc. But the most commonly-used measure is probably the "quality" of the freshman class, as measured by standardized tests like the SAT and ACT. Average incoming SAT scores at University of Texas campuses, for example, look like this:


















The Austin and Dallas campuses are getting students at 1200 and above while the non-selective regional campuses like Pan American and Permian Basin are below 1000. This conforms with nearly any measure of prestige and status one could name: Austin is an internationally known, Research I, AAU institution with a multi-billion dollar endowment and a football team that was lucky enough to beat my Ohio State Buckeyes in the Fiesta Bowl last month, not that I'm bitter. (Although: "Colt McCoy"? Really?) Permian Basin has none of these things, and probably never will.

But SAT scores leave the question of college student learning unanswered. It's odd, the way we give colleges credit for how their student did on a test they took while they were juniors in high school. Colleges argue that high SAT scores are an implicit quality signal because they reflect high demand, but the demand may just be for the prestige and the football team and the nice facilities and the chance to hang around with other students who also have high SAT scores. To really get a handle on learning, it makes more sense to test a sample of freshmen and a sample of seniors, and see how they compare. And in fact the University of Texas system has done exactly that, using the Collegiate Learning Assessment. Here's what they found:





















Each block on the graph shows two data points: freshman and senior scores on the CLA. As you'd expect, freshman scores correspond fairly closely with SAT scores: Austin and Dallas have the highest, regionals like Permian Basin the lowest, and the rest are in between. Much more interesting is growth. While Austin students arrive at high levels, they don't seem to improve very much while they're in college--the difference of 53 points is less than half the national average of 111 points. This may because of some sort of "ceiling effect," or it may be that elite universities don't focus much on improving students who arrive in great shape to begin with. Pan American and Permian Basin have very similar freshman scores, but Permian Basin's growth is more the double that of UTPA -- 197 to 90, bringing students from well below the national average on entry to above it on completion.

The CLA, it should be said, is not the be-all and end-all of college assessment. It's a general assessment of analytic reasoning, critical thinking and communications skills that doesn't measure mastery of the disciplines. It's subject to measurement and sampling error, like any standardized test. But it's also being used at hundreds of institutions and is based on a lot of smart thinking in psychometrics. It should be the beginning of much more attention to how much students learn while they're in college. This is how we ought to be thinking about success and prestige in higher education.

The CLA results also highlight severe limiations in the way we credential college students, and the vast differences in ability among students who are all pushed through a system that in many ways assumes they're the same. Note that despite the unusual growth at Permian Basin, seniors there still score well below freshmen at Austin. The premium given in the job market to degrees from highly selective institutions is, in that sense, quite rational; students could literally learn nothing while at an elite college and still outperform most other college grads.

The real inefficiencies and failures in the labor market occur at the individual level, particularly among the great masses of students with degrees from non-selective and thus largely undifferentiated instutitions. Lets say you're a very bright student who, for financial or family reasons, chooses to attend college at a local four-year institution like UTPB, except in a state that doesn't publish value-added measures like the CLA. You work hard, excel at your studies, and graduate at the top of your class. Do you get credit for this? No, you do not. The market cares little about college grades because they're opaque and inconsistent. So it assigns you the average value of a UTPB freshman, based on SAT scores, because that's the only comparable information it has. By the same token, the guy who finished last at Austin is over-valued in the market. And of course the brilliant person who never got a college degree at all is left completely out in the cold.

Sample-based measures like the CLA are only the beginning; what we really need to do is start attaching a lot more useful information to individual college credentials while also making the credentialling process itself more open and flexibile, less about having been taught by some kind of formal institution and more about having actually learned something real.

"No, we don't cheat. And even if we did, I'd never tell you."

Tommy Lasorda was talking baseball, but there are edu-implications. Gotham Schools is covering some back and forth about the process of grading the New York state ELA (English Language Arts) assessments. Eva Moskowitz says it’s easy and shouldn’t take so long, but teacher/grader/blogger “Miss Brave” says it’s disorganized and potentially unfair. Scoring problems are not new, but Miss Brave highlights a particularly tricky problem:

If you and all the other graders at your table happened to notice that the essay appeared to be written in two very different handwritings, as if it sure looked like the teacher had made a few changes, and you voiced your concerns, your
objections were dismissed.

Having teachers proctor high-stakes exams in their own schools is an open invitation for problems, especially where cash incentives for performance exist. Of course, we hope that teachers don’t influence the answers of their students during a test, and undoubtedly most of them wouldn’t. But when a student is struggling, and a teacher is walking by, it can be difficult to resist the urge to help. And even a pause or a quizzical look can influence a student’s answer. Unfortunately, it seems the fox is guarding the hen house on this one. Principals, districts and states have little incentive to check for this particular kind of score inflation, unless it becomes so egregious that it can’t be ignored. As for Miss Brave? She was told to “MYOB.”

The Growth in Growth

Prior to the 2005-2006 school year, schools around the country were required to count how many students in each school were "proficient" to determine whether a school made Adequate Yearly Progress (AYP). That year, two states, Tennessee and North Carolina, began the Growth Model Pilot program to allow schools to meet AYP another way, by moving children sufficiently along on a path to proficiency. No more was it just about status; growth and progress mattered too. It was a fairly inauspicious start: in that first year, a grand total of seven schools nationwide made AYP by growth targets alone. But in 2007, 353 did, and last year, 1,571 schools did it. This represents just a fraction of our country's 90,000 schools, but the growth in growth is a positive sign.

The difference between old and new is one of "status" versus "growth." Status models count as making AYP only those students who score above a pre-determined cut-off. Schools have a strong incentive only to worry about those students close to meeting (or close to failing to meet) that bar. These are the "bubble kids," the ones where extra attention may be just enough.

In contrast, the best growth models encourage adults to watch the progres of all students. If a student fails to meet proficiency standards but makes sizeable gains in the process, that student counts towards a school's proficiency. In Tennesee, where they have a particularly rigorous growth model, a student who scores in the "proficiency" range is not counted as such if she has slid backwards to the point where her future proficiency is in doubt**. No more ignoring kids at the poles.

The growth models as currently implemented are by no means perfect. Several states, for example, have implemented a system where students have to cross thresholds into newly created tiers to qualify as making growth. These are more or less arbitrary and not much more than additional layers of status--a student could still meet AYP by making a small jump across a threshold or fail to make AYP despite large gains within a tier. As more states get approved to use growth models--15 will be using them in 2009--further study will be needed on their impacts (see the first program evaluation here). Yet, growth models are already adding needed flexibility to the unpopular No Child Left Behind Act, flexibility that may better identify successful schools that help students below proficient make larger than expected gains.

**To clarify, I meant this sentence to suggest only that Tennessee's growth model was rigorous in this specific context. Whereas North Carolina and other states count all kids above the target plus those on pace to be above it as making AYP, Tennessee adds another wrinkle. It looks like this:

AYP = kids above proficient + kids on pace to be proficient - kids above proficient on pace not to be

This is just one among many trade-offs each growth model makes. North Carolina, for example, chose a relatively simple model that takes the amount a student is behind and divides it by three or four years, depending on the student's grade. If the student made gains higher than that, they count towards AYP for growth. Tennessee, on the other hand, uses multiple regression to predict future scores, a far more complicated and difficult to explain system. These issues are complicated, and the implementation choices states make matter. Look for more from Education Sector in the near future on this very topic.

The Americans

One of the great pleasures of living on Capitol Hill is the ability to walk out the front door on an unseasonably warm Sunday afternoon in February and amble down to the National Gallery of Art. Great national museums are normally experienced during short vacation-related timeframes that force you to engage in a fairly brutal calculus of weighing the desire to see iconic works against the desire to fully appreciate them and your brain's limited capacity to process the experience. Living nearby means you can focus on small pieces of the permanent collection or ignore it entirely and give all your attention to exhibitions like the one currently on the ground floor of the West building dedicated to Robert Frank's seminal book of mid-20th century photography, The Americans.

Before today, I knew enough about art and culture that I could have told you that The Americans was considered to be a milestone in photography, that Frank was a European Jew who brought an outsider's perspective to America, much as Tocqueville had, and that the book was highly controversial on publication due to its unsparing portrayal of race and class in the 1950s, which stood in stark contrast to popular photography like that published in Life magazine. Knowing things like this serves you in good stead at dinner parties or if you happen to end up on Jeopardy or Who Wants to Be a Millionaire. It's also no substitute for spending a couple of hours actually looking at the photographs, thinking about what they mean, and marveling out how Frank condensed years of work into less than 100 images, each of which contains a small world and which together seem to miraculously tell a story as deep and rich as the nation itself.  Yesterday I only knew about Frank's work, while today I know it, or at least I've begun to know it, and that makes all the difference.