EVENT: Computers, Professors, and the Cost of Higher Education

Note: Pushing this back to the top as a reminder to sign up, and also because the magazine article in question is now live to the world, here. Read it! Right now!  

It's commonly believed that higher education suffers from an acute case of "Baumol's Cost Disease," an affliction that causes labor-intensive industries to become less productive over time. It takes a professor just as long to deliver a 90-minute lecture today as it did 100 years ago, the thinking goes. But other industries have radically increased productivity in the meantime, often via use of technology, driving up the cost of highly-skilled labor across the labor market. In other words, colleges have to pay a premium for productivity increases they don't actually get, which is why tuition is now a zillion billion dollars and anyone planning to send their kids to a private college should expect to be in in debt for the rest of their, and their children's children's, natural lives. 

In fact, this is wrong. Colleges are perfectly capable of becoming more productive in the same way that lots of other industries have--by substituting capital for labor and replacing lots of expensive employees with less-expensive computers. It's not a theoretical idea--hundreds of colleges are doing this right now and more are jumping on board every day. Given that colleges are about to be mightily screwed by recession-induced budget cuts, this trend is likely to accelerate. But you haven't heard about it, in part because the significant cost-savings aren't being passed on to consumers in the form of lower prices. This is the subject of new article I've written for Washington Monthly which you can--nay, must--read here. 

And to see a sure-to-be-fascinating discussion of these issues, live and in person, featuring a panel full of experts, plus myself, sign up for and attend this Education Sector Event, to be held on the morning of December 2nd here in Washington, DC. Ask questions, heckle, throw things, get free magazines and related materials--it's sure to be a great time. 

Philadelphia Public Schools Gain Market Share, Blames Charters

In the private sector when an enterprise gains customers it's a good thing. Apparently that is not the case in Philadelphia School District, where their chief budget officer says charter schools are costing the district $105 million because 27% of their students were previously home-schooled or educated in a private school. Putting aside the idea that a 5% increase in market share is a bad thing, let's do some math. Stay with me here.

The city's chief budget officers claims charters, because of increased market share, are costing the city an extra $105 million. Charters educate 34,4000 students in the city and receive $320 million in reimbursements (including some state funds) for a total per-pupil expenditure (excluding private money) of $9,302.33.

The city reimburses charters $8,088 for every student in general education and $17,658 for every student in special education. Assuming charters took the same percentage of special education children as traditional public schools (13.2% in Philadelphia), how much should the city be spending on these new charter students?

13.2% of children in special ed. * $17,658 per student = $2,330.86
86.8% of children in regular education * $8,088 per student = $7,020.38
$2,330.86 + $7,020.38 = $9,351.24

So charters are actually getting less than they should. More students, less money, and the city complains?

Bailout Back and Forth

Treasury Secretary Henry Paulson set off a flurry of activity when he suggested last week that some of the $700 billion bailout money might go to help companies issue private student loans. Higher Ed Watch explains in this post why Paulson's plan is a bad idea.

NASFAA (the National Association of Student Financial Aid Administrators), though, seems to think it's a great idea, saying that "stricter loan eligibility requirements and higher interest rates and fees on non-federal loans are jeopardizing educational opportunity." A letter sent Wednesday from representatives of colleges, students, and organizations like the Project on Student debt explains why NASFAA is wrong, and Higher Ed Watch explains what colleges can do to ensure educational opportunity without any government bailouts. Inside Higher Ed has all the back and forth here.

But while everyone has been talking about private loan eligibility, one of the biggest, most genuine threats to college access--reductions in public college funding--has actually been happening. The California State University system got approval on Wednesday to turn away at least 10,000 eligible students next fall because of overcrowding and underfunding.

Cash-strapped Colleges

With headlines predicting a steep recession, colleges and universities are already making budget cuts. Compare what some schools are saying about the financial crisis:

Morton Schapiro, president of Williams College in Massachusetts, which has long had a commitment to accepting students without considering their financial situation, said he doubted that all colleges with such full need-blind policies would be able to hold to them.

“The major dial you turn for most financial crises is that you admit more students who can pay, as a way of increasing revenues,” Mr. Schapiro said. “With the tremendous decline in wealth, I think fewer people will hold on to needs blind.”

with what others are already doing:

In October, Cornell University in Ithaca, N.Y., increased its fund-raising campaign goal for undergraduate scholarships to $350 million from $225 million, which has "helped reinvigorate giving" specifically for this priority, says Simeon Moss, press office director at Cornell. While overall giving is down, donations directed toward undergraduate aid have soared to $63.4 million in fiscal-year 2008, from $13.7 million in fiscal-year 2007, he says.

There are good and bad ways colleges and universities can manage their budgets during tough times. Tying fundraising efforts to student financial aid seems to be one of the good ones. Kudos to Cornell for taking proactive action.

Stalking the Iron Sheik

Last night I saw an advance screening of Darren Aronofsky's new movie, The Wrestler, starring Mickey Rourke as an aging, down-on-his-luck professional wrestler and Marisa Tomei as a local stripper who may or may not have a heart of gold. It's a very good movie that you should definitely go see when it's released in theaters next month. The Wrestler is a human drama first and foremost, but it's also about the cold reality of working life, the aesthetics of late-80s hair metal (musical and otherwise), and the strong bonds of subculture brotherhood. It was shot on a shoestring $6 million budget in hand-held documentary style, and the contrast with the precise, controlled visuals of Requiem for a Dream and the criminally-underrated The Fountain is pretty amazing. 

I'm not giving much of the plot away in revealing that the movie revolves, in part, around the possibility of the protagonist, one Randy "Ram" Robinson, participating in a 20th anniversary rematch of an epic 1989 pay-per-view showdown with a wrestler named "The Ayatollah." As we all know, this is a thinly-veiled reference to the classic 1984 title bout between Hulk Hogan (Randy basically looks what Hogan would have looked like if he'd spent the last 20 years on a crystal meth bender, or living the life that Mickey Rourke actually lived) and Hossein Khosrow Ali Vaziri, a.k.a The Iron Sheik. And this, in turn, is all the excuse I need to tell my Iron Sheik story.

The Sheik, now retired, was born in Iran and had real wrestling skills, competing on the national team (Iran has some of the best wrestlers in the world) in the late 1960s. By the 1970s he had moved to the U.S., where he wrestled under names like The Great Hussein Arab (insert joke about McCain campaign rallies here). Certain contemporaneous developments in geopolitics made his status as the designated "heel" or bad guy in the pro wrestling ring more or less inevitable, and he went on to a successful career, winning the heavyweight title before losing it to Hogan while awing fans in mid-sized regional municipal auditoriums across the land with the power of his patented finishing maneuver, the "camel clutch." 

Anyway, it was some time in the late 1990s, and I was in the Indianapolis airport waiting for an outbound flight. I was hungry so I went to the scary food court, grabbed something greasy and marginally edible, and went to find a table--only to see, sitting right in front of me, the Sheik himself, amiably chatting with another really large dude who was obviously a fellow wrestler. They'd been in town the previous night for a match at the since-demolished Market Square Arena (site of Elvis' last concert, fyi). I didn't want to interrupt them--the 12-year old in me still found him extremely scary--so I sat nearby to eavesdrop. Their conversation went something like this:

Another Really Large Dude: "Hey, did you hear that The Undertaker bought a place on a golf course in Myrtle Beach?"

Sheik: "Real estate is always a smart place to invest."

ARLD: "Yeah, Hacksaw Jim Duggan told me he cleared $100,000 on his place in Florida last year."

Sheik: "If Jake the Snake had at least put his money in T-bills like I told him to, he could have retired five years ago."

And so on and so forth, the most mundane conversation imaginable, except they referred to everyone, and each other, by their stage names. It was surreal. Eventually they finished their burgers and walked to airport security, at which point the security guys started jumping and down, pointing, and yelling "the Iron Sheik!" (airport security jobs were close to minimum wage gigs back then, as you'll recall). We ended up on the same plane, but the Sheik and ARLD were in first class so I didn't get a chance to hear more.

No real point to this story other than a brush with semi-greatness, and you should really go see The Wrestler.  

Dear President-Elect Obama...

NCLB reauthorization may not be at the top of the next administration's to-do list, what with the economic meltdown and two wars, but President Obama will need to tackle President Bush's signature education law eventually. And he'll need all the good ideas he can get.

Today, ES releases two briefs offering ideas on how the Obama administration can reform NCLB. Title 2.0 focuses on how President Obama can revamp the federal role in human capital by focusing Title II dollars on higher impact reforms. And In Need of Improvement offers a number of steps Congress and the Obama administration can take to strengthen NCLB's choice provision.

Not-So-Great Expectations

Higher education is haunted by a formula, which goes something like this: P/R=G.

The P stands for student preparation, broadly defined — the combination of innate ability and elementary-school and secondary-school preparation that students bring to college. Imagine those attributes normalized on a scale going from 0 to 1, with 1 describing the smartest, most well-educated student in the world.

The R stands for rigor, defined by individual colleges and universities — academic requirements, placement-exam cut scores, and the general difficulty of the work. R increases as standards become more rigorous, with the top values at places like the California Institute of Technology and the Massachusetts Institute of Technology.

The G stands for the odds of a student's earning a degree. An intelligent, well-prepared student attending a college with typical standards would be very likely to graduate. An ill-prepared student who enrolls somewhere with unusually tough standards would not.

The P/R=G formula dominates the way people think about college graduation rates and student success. And, not coincidentally, it puts colleges in the position of having no real responsibility or efficacy when it comes to making G higher. They can't make P higher, because raw ability is what it is, and the elementary and secondary schools are someone else's problem. And they can't make R lower, because that would betray their scholarly ideals and dumb things down for the best students. A low G is regrettable, but really, what can be done?

It's a pretty depressing conclusion. So I was glad to read a report on the Community College Survey of Student Engagement, known as Cessie, which says that the formula is all wrong. 

Read the rest of this column at the Chronicle of Higher Education here. 

Hot Boys (With Audio)

If you missed Education Sector's talk last week with author Peg Tyre you can listen to it below. The discussion features Tyre, best-selling author of The Trouble with Boys, New America Foundation Senior Analyst Sara Mead, and USA Today columnist Richard Whitmire. Topics ranged from gender differences in NAEP scores and ADHD diagnoses to boy-girl brain differences and why boys like the video game World of Warcraft.

Last Week, Next Up on Testing

The paper we released last Monday is the first in a series to explore what the next generation of assessments might look like. I received many many emails and comments over the past week, some via our online discussion. A quick recap from all of that:

First, there is a surprisingly strong reaction to the term “21st century skills.” Most wanted to say something about how important and significant these skills are to the students they teach or know. And in general there was agreement that there is a set of “21st century” skills that students need more now than before. But the term “21st century skills” seemed like an unpromising default to many, a way of avoiding specificity. One comment was particularly pessimistic but perhaps a fair point: “It’s a meaningless term—by the time we figure out what it means, it will be the 22nd century. Then what?” As an aside, I initially avoided the term but decided to take it on to see if I could push past the platitudes.

Second, there is a lot of interest at the local level in the new assessment tools and quite a few questions came in about which ones worked best, which ones should teachers/schools/districts use. My response is that there isn’t one tried and tested (sorry for pun) assessment that districts and schools should adopt and start using. The CWRA, which I profiled in the report, is one example—and I think a good one—of how schools are trying out new forms of assessment that measure reading, writing and math skills and problem-solving, inquiry and decision-making skills. The larger point is not that this is the right test for every school or district, but that this is the right direction for assessment.

Related to this is the problem of cost. There was a lot of concern about finding funds for assessments like the CWRA. “Even if my school wanted to try this,” wrote one teacher about the CWRA, there’s no way they would spend any extra money on it. This isn’t surprising, wrote Jack Beirwirth, who is the superintendent of Long Island’s Herricks Public Schools, the first public school district to use the CWRA. We’re all cutting back, he said, on programs, services and jobs. But the CWRA is well-worth the extra cost--teaching and measuring critical thinking and analytical reasoning are among our goals, he explains. Herricks also became one of four school districts participating independently in the 2006 PISA.

There were also a lot of comments about breaking down the distinction between instruction and assessment. Can we “embed” assessment in teaching, so that teachers can learn how their students are doing and improve their practice at the same time. So that assessment is not seen as a series of burdensome tests but also a tool for continuous learning. This isn’t easy—it requires teachers who know how to use assessment both for generating summative information and to inform their daily practice. But it can be done--see Paul Curtis’ description of New Tech High's approach.

Emerging technologies play a big role in this (I received several emails asking “what about technology?”). Our next paper on assessment--this one by Bill Tucker-- will examine how information technology can be used to improve assessment.

A Financial Aid Shake Up

Last Friday, FastWeb, a free, online scholarship search service, released the results of a survey it conducted on student borrowing, showing that half of students applying for private loans, parent PLUS loans, and home equity loans were denied access to funds. If this number is accurate, it means that a large number of students will need to adjust their college choices--from a private to a public institution and from a 4-year to a 2-year institution--because of financial constraints.

There's good reason to be skeptical of FastWeb's high numbers--the survey was sent out to users of the website (people already looking for additional financial aid) and the response rate was low (1,202 responses out of 7 million invitations). It's likely that those who took the time to fill out the survey were also the ones having the most trouble finding financial aid.

But we do know that private student loans, because of the tightening of credit markets, are more difficult to get. While the actual percentages of students being denied private loans may not be as high as FastWeb reports, it's still higher than in previous years, when easy access to credit helped fuel a boom in private lending.

So what are policymakers to do?

Tuition levels have risen to the point at which federal lending limits are insufficient to cover tuition at pretty much all private colleges and even some public, 4-year institutions. As a result, some congressional members are talking about raising federal loan limits again to make up for some of the lost private loan dollars. But that would be a mistake.

Tuition levels were able to get as high as they are partly because of easy debt--in an era of loose borrowing requirements, students were able to access large amounts of private loans. This was a great situation for colleges--students took out large amounts of loans, the institutions got paid, and students were left to bear the debt burdens. While lip service was paid to the need to reduce tuition prices, there wasn't much real pressure - enrollments stayed high and tuition bills were paid. Now that private lending is more limited, there may be some real price pressure on colleges to reduce tuition rates, or at least limit increases.

Adding to the argument against raising federal loan limits is a recent admission by the University of Phoenix that it sets tuition partly based on federal loan limits. And it actually cut tuition in its two-year Axia college division after seeing that students were dropping out because they maxed out their loan eligibility. So maybe less borrowing and more pressure to lower prices isn't such a bad thing in higher education, and could lead to more reasonable tuition rates.

Even when the credit markets loosen up, we (hopefully) won't return to the wild west days of lending that led to our current problems. Just like subprime mortgates aren't really good for home ownership, subprime student lending isn't really good for college access or success. Tighter private loan markets might result in a more cost-consciousness, on the part of students and parents, and institutions.

Stop the Violence

Last Sunday, the Washington Post ran a story about the recent surge in school violence at Hart Middle School, where, in the past several weeks, three teachers were assaulted, a 14-year-old was charged with carrying a shotgun and students discharged fire extinguishers throughout the building.  Hart, a school currently placed on “restructuring” status under NCLB due to its repeated failure to make adequate academic progress, has made little headway in its attempt to raise achievement despite the installment of a young new principal and the promise of additional interventions to support students and families.

Little headway, that is, until a few day ago, when DC schools’ Chancellor Michelle Rhee fired Hart principal Kisha Webster, supplanting her with a central office administrator. In recent coverage of this sure-to-be controversial firing, the Post gives Ms. Webster’s take on the firing.  Claiming to have been “set up,” Ms. Webster blamed her downfall on a lack of resources, second-tier teachers, and an unsupportive central office. She then went on to assert that publicity led to her demise. She is quoted here as saying: "If I had been able to keep things quiet, I'd still be [at Hart].”

Never underestimate the power of a headline. It seems, at least in the case of Hart, DCPS is more responsive to external coverage than internal warning signs and threats of disaster. From what I can gather, the problems plaguing Hart had taken root long before Webster took over in September. And DCPS knew of Hart’s struggles and should have done a better job of supporting the school and its staff from the beginning, rather than waiting for a Post exposé to prompt swift action. Regardless of central office’s failure to intervene, as school leader, Webster had a responsibility to create a safe atmosphere and climate of learning. And it is apparent that in her short tenure, neither goal was accomplished.

Hopefully, the lesson learned from Hart is that DCPS can no longer afford to reactively put out fires across the district. A change in principal may have the desired effect of ending the recent surge in violence, but unless it is coupled with better policies and practices, we’re unlikely to see systemic improvement. A well-developed and thoughtful set of policies should be crafted and put into place in each school, accompanied by a cadre of prevention programs.

Unlike a decade ago, we now have considerable evidence for successful, innovative school-based prevention programs that enhance the ability of schools to function properly, and, in turn, reduce incidences of violence and discipline considerably. See here and here for some good work being done on this front. Let’s not wait for the next Post headline to begin exploring these programs and implementing them in our schools. Instead, let’s think about some ways in which we can use the evidence base to inform both and practitioners and policy makers. 

The research around school-based prevention programs suggests that more important than specific programs or curricula are the principles upon which effective strategies are based and the fidelity with which they are implemented.  Schools can decrease problem behavior by organizing and managing themselves effectively, creating environments that support prosocial behavior, instituting clear rules and expectations, and creating structures and supports that help administrators, teachers, and students work together to meet those expectations. Yet these supports are often symbolic (e.g. behavioral contracts) or entirely lacking in the most troubled districts. It is encouraging to know that in some districts (like D.C., according to a draft five-year action plan), strategies are underway to bolster Student Support Teams (SSTs) to better coordinate academic and behavioral interventions for at-risk students. I’d like to see SSTs expanded to become a more universal approach, especially in urban districts where the vast majority of kids can be categorized as at-risk.

From a policy standpoint, more dollars should be appropriated for school-based prevention – despite the relative cost-effectiveness and proven success of school-based programs, they only receive an estimated $6-$8 per pupil per year in federal expenditures, not enough to be widely and properly implemented. Federal dollars should be targeted towards high-need districts in low-income, urban communities that exhibit a disproportionately high rate of violent and disruptive behavior and yet have the least capacity for solving such problems. And because research suggests that the quality of implementation is at least as important as the type of program, monitoring and technical assistance to districts and schools should be ramped up. 

- Posted by guestblogger Sara Yonker

Dispatch from the Front Lines

Policy types debate whether school systems should try to fix failing public schools or close them and replace them with charters or other types of new schools. Answer: it's a lot easier to create a constructive school culture with a new team than having to change the hearts and minds of existing staff. But sometimes the reality on the ground, to use the cliche, makes school reform a little trickier. In Chicago, school authories have opted for trying to turn around six failing high schools rather than closing them down because moving students to new schools would cause them to cross gang boundaries.

Secretary Spellings, Blogger

Margaret Spellings, the United States Secretary of Education, is guest-blogging today at our fellow institutional blog, Eduwonk. (She must have missed the memo that bloggers are all a bunch of unshaven, pajama-wearing D&D players living in their parents' basement.) Education Sector is, of course, very pleased to host her discussion of education and technology. On a larger note related to all the current discussions of the new administration's education policies: I think the Secretary and her staff have done some important work over the last three years in the higher education realm and I hope it continues in the future. 

Perspective

Alan Odden says that Michelle Rhee's proposal to give DC teachers the option to trade job security for a lot more money, if they want to, or not, if they don't want to, "would raise eyebrows everywhere, because that would be a gargantuan change."

The only real eyebrow-raising element of this is that it's national news. Here you have someone who was brought in to reform an organization that everyone--everyone--agrees was terribly dysfunctional, and her response has been, in part, to identify the employees who aren't doing a good job and replace them with better employees. This isn't just obvious and rational; it's ordinary. It happens all the time, particularly in organizations that are labor-dependant. And that's a good thing, because we're all much worse off when low-performing organizations stay chronically unreformed and low-performing. 

Which raises a broader point: from a p.r. standpoint, taking a stand here and now on this issue is a terrible decision for the American Federation of Teachers.  Teachers have a very good case to make when they say that, compared to other professionals with comparable levels of education who work at similarly difficult jobs, they're underpaid, disrespected, and forced to work in conditions that most people wouldn't tolerate. Fair compensation, dignity, and a basic quality of working life are things that all people understand and have a right to expect in their jobs.

Permanent job security regardless of performance, by contrast, is enjoyed by a small and shrinking portion of the workforce. Most people can't relate to tenure. They've never had it, they never will, and they understand why. Drawing the line there is a strategy of alienation for a group that has fewer friends than it understands and fewer than it needs. 

The Myth Continues

Nicholas Kristof's column today urges President-elect Obama to move education to the front of the policy line. We're all for that here at The Quick and the Ed. But Kristof commits a major sin by repeating the line, that we've twice debunked, that the current generation is the first to have less education than their parents. The line is false both in rhetoric and fact.

I understand Kristof's pain. He's trying to create a crisis where one doesn't exist so as to transfer energy into an important issue. Realistically, our schools as a whole aren't bad; they're just kind of...flat. Take some time to look at the original data where the claim comes from, and here's what you'll find:

• the United States had and continues to have a very high percentage of its adults completing a higher education degree.
• while that percentage has crawled upwards here in the States, other countries have cleared it. Where once we were first, we're now tenth.
• our high school graduation rate has fluctuated between 70 and 75 since at least 1995. Some years are better, some are worse.
• many of our adults acquire GEDs over time, boosting the percentage of the population with a high school diploma. That rate hovers around 87 percent for all generations.
• college graduates in the US continue to have success finding employment for themselves, creating it for others, and capitalizing on their credentials to boost income.

The scariest statistic is not the one Kristof repeats. It is that we have one of the highest rates of college entry, but we have yet to figure out how to get them through college:

Only 54% of entrants to higher education in the United States obtain a degree. Along with New Zealand, this is the lowest survival rate among OECD countries, where the average is 71% and as high as 91% as in Japan.

Our Comics-Loving President-Elect

As many others have noted, the Huffington Post reports that president-elect Obama "collects Spider-man and Conan the Barbarian comics." This is, of course, welcome news for right-thinking people everywhere. But it also raises more important questions that it answers. Which Spider-man and Conan the Barbarian comics? Some are pretty easy to guess--all Conan comics begin with the classic 1970s Roy Thomas / Barry Windsor-Smith run at Marvel, which arguably remains unequalled. Kurt Busiek's 2003-2006 stint for Dark Horse also stands out. But there were a lot of comics in between. Was Obama picking up the oversize, black and white Savage Sword of Conan back in the 80s? Is he re-reading old issues of King Conan for insights on how Conan dealt with the heavy burden of leadership after finally ascending to the throne? James Earl Jones as Thulsa Doom: inspired or ridiculous? These are important things to know. Similarly, will our future leader do the right thing and officially denounce the Spider-clone as the most egregious retcon in all of comics history? He's old enough to remember when it was big news that Peter Parker married Mary Jane (only to be un-married earlier this year in another retroactive continuity re-boot, ugh)--can he honestly say that his position on that has been consistent over the years? Did he publicly denounce the black suit, on the record, when it was first introduced post-Secret Wars, or did he only follow popular sentiment later on? Todd McFarlane goosed sales, but did he ultimately leave the character better or worse off? I assume he's pointing Malia and Sasha toward Ultimate Spider-man, if not an explanation on his thinking would be helpful. Frankly, the fact that we don't have answers here suggests a breakdown of the mainstream media's pre-election vetting process. 

All Deliberate Speed

From a New York Times article about a small Lousiana high school that may soon cease to exist:

This tiny, rural town in Cajun country is struggling again to find its racial equilibrium. A 43-year-old desegregation case remained unresolved on Nov. 4 as voters narrowly rejected a property tax increase to build a new Ville Platte High, which has faced decades of neglect since white flight accompanied integration in 1969.

President-elect Obama's historic victory has been frequently described over the past week as the culmination of a long struggle for civil rights that began in the 1950s with historic court decisions like Brown v. Board. And there is great truth in this. But it's worth remembering that, 54 years and counting since the Supreme Court ordered schools to be desegregated "with all deliberate speed," we're still not finished desegregating. Promises made to children in the middle of the 20th century remain unfulfilled for their grandchildren. There's a whole division within the Justice Department full of lawyers who travel to areas, mostly in the rural South, where educational opportunity remains divided along racial lines. There is still a lot of work to be done. 

Magnitude

Expanding on something I alluded to in the post below about training and certifying teachers vs. attracting and selecting them: the way these issues are often discussed reveals one of the weak spots in the way research is applied to policy. Essentially, people don't properly account for differences in magnitude.

All rational people want education policy to be informed by the results of objective research and inquiry, conducted using transparent, widely-accepted methods. (In a lot of ways, this is just a restatement of the definition of "rational person.") The kind of analysis that informs policy tends to involve testing hypotheses and identifying differences. Consider randomized control trials, the "gold standard" of research, which increasingly are being used in education. You randomly assign subjects to two groups, apply a treatment to one of them, and see if some pre-defined results vary. If the variance is large enough to meet agreed-upon levels of statistical significance, you conclude that the treatment had an effect. The same basic approach, even without the randomized assignment, is used to tackle other questions--like, for example, the effect of certification on teacher effectiveness.

So let's take another look at that chart with the three overlapping bell curves, one showing the distribution of teacher effectiveness among certified teachers, one showing alt-cert teachers, and one showing uncertified teachers.

If you squint your eyes and look carefully, you'll note that the solid line--the certified teachers--is slightly to the right of (with teachers increasing in effectiveness as they move from left to right along the scale) the other two. In other words, the population of certified teachers was slightly more effective. If the populations were large enough--if, say, they represented every teacher in America--that difference might be statistically significant. It might even be significant beyond a shadow of a statistical doubt, at the .0001 level or what have you. And if that were the case, the results of the research would very likely be translated for lay readers and policymakers into something like this: "A study from [fill in the blank] found that certified teachers are more effective than non-certified teachers." And the natural policy response would be that we should make sure that all students are taught be certified teachers--they're better, after all--and we should invest in and strengthen certification programs, since they work.

The problem, of course, is that if you put your common sense hat on and think for 30 seconds about what that graph actually shows, you naturally conclude that such a policy response would be insane. The difference between certified and uncertified may indeed be statistically signifcant, but it is a tiny statistically signficant difference, one that pales in comparison to the extent to which teachers vary within those populations, and the extent to which the populations are the same. To state the obvious: the most important differences tend to be large, not small.

The vernacular of research and policymaking, in other words, is insufficiently precise about and sensitive to differences in magnitude. All significant differences are not equally signficant just because they met the same test of signficance, but they are often thought of as such.

A lot to Say About Testing

We have a lot of comments to the testing and 21st century post from Friday. I'm also getting a lot of emails on this--from teachers who say they are already teaching "21st century skills" and what's new about this anyway?, from district officials saying it's too expensive to assess these skills (isn't it?), from content and curriculum specialists concerned that the focus on 21st century skills will distract from efforts to teach reading and math, and many others. Tag cloud to the side is thanks to Dennis Richards, who wants to know what kinds of teachers and teaching tools it will take to teach these skills to all kids. Much more on all of this at the discussion here. Join in: ask questions, post comments.

g(t)?

Writing in the Boston Globe (per Matt Yglesias), Harvard economist Edward Glaeser cites Tom Kane's research on teacher quality, saying:

The first step toward improving teacher quality is to attract more talented teachers. The second step is to improve teacher selection on the job, promoting the best and encouraging the worst to help society in some other way.

The key words are "attract" and "selection." But just as important are the missing words: "certify" and "train."

The conventional system for bringing teachers to classrooms relies on pre-service training and certification to ensure quality. Students are required to complete a state-approved training program, usually offered at a university, and then obtain state certification, which increasingly involves passing some kind of exam.

Tom Kane's research shows that the training and certification model is near-worthless as a means of effectively guaranteeing or differentiating teacher effectiveness. The paper he wrote with Robert Gordon and Douglas Staiger for the Hamilton Project in 2006 is well worth reading in full, but you can get the gist just by looking at Figure 1, on page 8.

We have three bell curves, one made of a solid line, one a dashed line, and one a dotted line. Each of the lines shows the distribution of a group of elementary teachers in Los Angeles based on classroom effectiveness, as measured by their students' math test score gains, after controlling for baseline scores, student demographics, and program participation. The solid line shows the population of traditionally certified teachers, the dashed line alternatively certified teachers, and the dotted line uncertified teachers.

The graph shows two crucial things. First, the bell curves are wide--there is a lot of variation in effectiveness among teachers within each population. Students with the most effective teachers gain ten percentile points or more in one year; students with the least effective teachers lose ten percentile points. To put that in perspective, the cumulative black - white achievement gap nationwide is about 35 percentile points.

Second, the curves are, for all intents and purposes, exactly the same. There's a tiny bit of daylight between the certified teacher curve and the other two. But it's dwarfed by vast differences in effectiveness within the populations. The long, tedious argument within the education community about the virtues of training and certification amounts to debating how many angels can dance on the head of a pin. As near as anyone can tell, there is no way to figure out ahead of time who is going to be an effective teacher and who is not via traditional training and filtering processes. The best way--arguably, the only way--to figure who is an effective teacher turns out to be letting people teach and seeing if they're effective.

The logic of this quickly leads to a very different approach toward teacher quality--not train and certify, but, as Glaeser writes, attract and select. Do whatever you can to attract smart, motivated, talented people into the profession, and then impose a ruthless screening process based on results in the first few years. That doesn't mean you shouldn't try to make those first years as supportive as possible. Yet, seen through the lens of those overlapping bell curves, recent disappointing results from a study of induction models aren't surprising. Some students will suffer in the classrooms of the least effective teachers (academically at-risk students should be protected), but that's just as true of the current system.

The interesting thing is that the Kane / Staiger / Gordon paper came out two-and-a-half years ago. I'm not aware of any serious challenges to the data, methods, or conclusions. As far as education policy papers go, it received a lot of attention, getting written up in the Wall Street Journal here and being name-checked by Nicholas Kristoff here and Bob Herbert here. And of course it garnered hyperbolic praise on this blog, way back when.

Yet the education world keeps right on going with the traditional system. There's a lesson here in human psychology: when evidence is revealed that pushes people completely outside of their established frame of reference, a little mental "na-na-na-na-na-" device turns on and they simply proceed as if it doesn't exist. I suspect the dissonance is particularly jarring for education professionals, since the implication is that, paradoxically, there are severe limits on the extent to which good teaching can be taught.

The findings--which are supported by a host of other teacher effectiveness studies--also raise an intriguing question: If variation in teacher effectiveness isn't attributable to anything we can currently observe, then what's driving it? Is there some kind of innate teaching ability, a sort of g(t)? That combined with more generalized qualities like motivation, intelligence, and the experience of having been well-taught?

I don't pretend to know the answer, but I know that the empirical justification for continuing the current way of doing business in teacher policy is quickly eroding, and that researchers would serve education better by focusing more of their attentions on exploring the source of large differences rather than small ones.