Teacher Experience, Effectiveness, Etc.

Edwize has a extra-long post from Leo Casey (is there any other kind?) responding to what I wrote a few days ago about the Aspen Commission, value-added measures of teacher effectiveness, and teacher experience.

In addition to finding my writing less than fully felicitous (dude--that's cold) Leo seems to think we disagree about the impact of teacher experience on effectiveness in the classroom. We don't. Leo is exactly right when he says:

...the learning curve in teaching is very steep at the start of a teaching career, when the novice is mastering the fundamental skills, and levels off after that initial 3 to 5 year period. From the viewpoint of mastery of the teaching craft, therefore, it makes sense to say that there is little difference between a 10 year and a 20 year teacher. But whether a teacher has 10, 15, 20 or 25 years of experience, what never disappears is the crucial differential between a novice teacher and an experienced teacher. An accurate description of the research would be that the gap between the skills of the novice teachers and the skills of more experienced teachers grows significantly through five years of experience, and thereafter levels off.

We agree; this is what research, common sense, and personal observation tell us. However, Leo is leaving out an important qualifier here, one that makes all the difference in discussions of teacher compensation. He's describing the typical trajectory of individual teachers. In other words, any given teacher is nearly always significantly better in Year 5 than he or she was in Year 1 (and not much better in Year 25 than Year 15). But that does not mean that all Year 5 teachers are the same. In fact, some teachers are pretty great from the beginning, and improve to become spectacular. Other teachers struggle mightily in Year 1, and while they may improve, they never become very good. The differences can be huge.

As a result, some novice teachers are better than other veteran teachers, making up for lack of experience with talent, dedication, training, and hard work. For what I suspect is a good example of such a teacher, just scroll down three posts on Edwize itself, to this post, titled "Are My Students Suffering Because I'm a Novice Teacher?" I don't know the answer for sure, but if I had to guess, I'd bet they're not. That doesn't mean this teacher won't get better, of course she will. But judging from this post, she's probably doing pretty well by her students already:

I have even admitted to family, friends and most importantly myself that I feel like this is my true calling. I enjoy the kids, I love sharing information, and this career allows me to be myself. In fact, my personality has been vital in forming a good rapport with my students and earning their respect. Still I can’t help but to feel guilty because I realize they are like my first-year “guinea pigs”; they are getting their education from someone who is focused and intelligent but not seasoned. I am an excellent writer, fast learner and able to relate to my students, all good qualities that work in my favor as a teacher. The students have admitted many times during class that they feel like they do “real work” and are “really learning something” in my class, which sounds great coming from those who, in my mind, matter most (the students). I reflect often on the things my students have said to me such as, “Miss, I like that you are always fair” and “I remembered what you taught me when I took my exam.”

Resources are finite. If we tie up large amounts of money in mandatory experience-based salary increases in order to retain all experienced teachers, as opposed to only good experienced teachers, then we have less money available to recruit and retain good novice teachers. Increasing salaries for effective younger teachers in this way would decrease the high turnover that Leo is rightly concerned about, and increase the odds that when those young teachers learn from veterans, they learn something worth knowing.

Leo might say there's no way to fairly determine who's "good," and "effective," but I've always found that argument to be singularly unpersuasive. After all, he himself tells us that he's "visited scores of high schools through the years," and has observed the benefits of experience first-hand. I assume he doesn't just infer that experience leads to effectiveness, but that he has instead observed effectiveness independently, as a distinct quality. If Leo can accurately observe large differences in teacher quality in his visits, why do we assume that the principal who works there full time can't do the same, particularly if they supplement those judgments with student achievement data? Clearly, there are examples of principals who aren't up to this task, either as a matter of competence or integrity, but that's an argument for getting better principals, not for an absolute refusal to base individual teacher pay on how well individual teachers do their jobs.

At the bottom of his post, Leo swerves into the standardized attack on standardized tests. Look--nobody thinks standardized tests provide perfect information, or even close to perfect information. But the grown-up question here is obvious: what level of imperfection can we tolerate, given the way the information will be used? (See Sherman Dorn for comment on this issue.) Leo clearly thinks that our current tests fall below that threshold. Okay, I'll bite: how good does student assessment information--from standardized tests or any other source--have to be before it would be appropriate for use in determining teacher salaries? 100% perfect and unassailable in every way? If not perfect, how good? 95%? Let's put a number on the table and then figure out what it would take to get there.

Unless, of course, not getting there is the point.

Empowering Kids with Neuroscience

I wonder what Charles Murray has to say about this story on NPR about new research indicating that students’ beliefs about their intelligence can impact how they do in school. Students who believed their intelligence could grow got better grades than students who believed their intelligence was fixed.

An interesting part of the study design is that they didn’t just use motivational speaking to convince students they could change their intelligence, but they actually taught the students how their brains worked – that their brains formed new pathways when they were learning and could actually change to increase their ability to solve challenging problems and learn new material. Understanding the truth behind intelligence and the brain can be a powerful tool, one that these kids will hopefully use throughout their lives whenever someone tries to convince them that they just can't learn something.

Check it.

Just got a note about the D.C.-based organization Teaching for Change about some of the book events they host. Not all of them are education related, but author talks are usually pretty interesting, so check out the event list and recommended readings here.

Algebra is the Tool of the Oppressor

Via Joanne Jacobs. Funny.

Crazy Colorado Tax Policy

This article in National Crosstalk is well worth reading, in that it illustrates what happens when the sensible, democratically-expressed desires of the citizenry to invest public resources in important things like higher education run up against mind-bendingly foolish, anti-democratic conservative tax policies like Colorado's TABOR law. The result: a pointlessly complicated, Rube Goldbergian higher education funding system that accomplishes nothing other than confusing everyone and wasting resources that could be better spent elsewhere. Seriously, you really have to read it to believe it, this is full-scale, through-the-looking-glass public policy craziness.

Size Matters?

To clarify this: I got nothing against "big ideas" per se, nor did I say I did. But I don't think "bigness" should be the primary goal of policy development. Lots of big ideas are bad, and when big ideas are bad, they're bad in a big way. Nor should we write off small ideas just because they're small. Sometimes an accumulation of small ideas has more impact than a big idea. I doubt Alexander's desire to see the next generation of good big education ideas exceeds mine; I'm actually working to try to help develop some of them (and, no, I don't think they have to come from Education Sector). But I think it's equally important to head off the next generation of bad big ideas that will inevitably come along. And most of the good big ideas that stick need myriad small ideas following behind them to help make them work.

That said, I do have something of a knee-jerk loathing for purely symbolic actions that politicians undertake to try to get credit for "doing something" or even "caring about" a problem, rather than actually, you know, doing something that might be difficult. And that's what legislation to make education a Constitutional right strikes me as. Leave aside that it's politically going nowhere and nobody even thinks it is. What on earth would it mean in practice? Wouldn't it just lead to a lot of "John Marshall has made his decision; let him enforce it" type stuff from states that wouldn't know how to get there even if they had the will to? Would it actually help states, school districts and schools do a better job in any way? Concerns like these are why most thoughtful observers and a majority of D.C. City Council members last year turned against a proposed D.C. Charter amendment to guarantee all D.C. kids a "high-quality" public education. Everyone knew it was a purely symbolic act that, if passed, would have amounted to a very sick joke on D.C.'s kids. Wouldn't the same arguments apply to the national proposal?

Aspen and Value-Added

Day two of the Aspen NCLB report saw good notices in the press from the Bush Administration and the Democratic committee chairmen on the Hill, Ted Kennedy and George Miller, and more strong objections from various teachers union representatives. While Joel Packer et. al. hit the broad strokes in the Post and Times, Leo Casey got a lot more specific over at Edwize. It's worth taking a look, because there are a lot distortions and flatly incorrect statements to parse. Leo said:

NCLB was exactly on point in insisting upon a highly qualified teacher in every classroom. Educational research affirms the absolute centrality of experienced, accomplished teachers to the education of young people, especially for students living in poverty and at academic risk.

The recommendations of the report would abandon the work that states have already done to improve the quality of teaching, set aside the work that remains to be done, and adopt a proposal which has never been successfully implemented – the use of growth in standardized test scores as a measure of teaching quality. State of the art research on ‘value added’ and ‘growth’ models of standardized testing has concluded that they do not have the capacity to provide accurate micro-level data for schools or individual teachers. In the absence of such refined tools and with the real world of standardized tests that are themselves substantially flawed more often than not, the use of such tests as measures of quality teaching will simply create massive disincentives for schools and teachers to take on the most needy and academically at risk youngsters who perform poorly on such exams – the exact opposite of what we should be doing in American education.

Let's take these in order (all bold type added):

Leo says "Educational research affirms the absolute centrality of experienced, accomplished teachers to the education of young people, especially for students living in poverty and at academic risk." On the first point, "experienced," this is a gross distortion of what the research says. There's a lot of research that says teacher experience matters. Absolutely none of it says that teacher experience is central to the education of young people. The research says (see here for a good summary, and here) that experience matters in the beginning of a teacher's career, up to at most 10 years, and then not afterwards. And even when experience does matter, it accounts for only a small fraction of all the variance among teachers.

In terms of the centrality of accomplished teachers, it depends on what you think that word means. If you mean the accumulation of credentials, like state certification or master's degrees, absolutely not. Most studies say master's degree don't matter at all, and the evidence on certification is mixed, with small effects. If, on the other hand, you mean "accomplished" in the sense of "consistently helping students increase learning, as measured by standardized tests," the evidence is a lot more persuasive--but that's the very conception of teacher quality that Leo is arguing against.

Leo criticizes the commission for wanting to "adopt a proposal which has never been successfully implemented – the use of growth in standardized test scores as a measure of teaching quality."

A) That's not true, Tennessee has been doing it over a decade.
B) To the extent that it hasn't been implemented more broadly, that's because people like Leo object every time someone tries. It's the chicken-and-the-egg strategy of obstructionism--stop people from doing something new, and then when other people try do it, say "you shouldn't try to do it, because nobody's ever done it."

Leo then cites a 2003 book from RAND, titled "Evaluating Value-Added Models for Teacher Accountability," as evidence against the approach the Aspen commission recommends--evaluating teacher effectiveness by using year-to-year growth in student test scores. This book stands as the most authoritative treatment of the subject that currently exists, and it's going to get referred to a lot in the coming debate. So let's be clear about what it does and does not say.

The RAND book provides a very thorough and critical look at the extant research and available methods for estimating teacher effectiveness using test score growth. It reaches the absolutely reasonable conclusion that none of the measures are perfect, and that anyone relying on value-added data should be aware of the many potential sources of bias and error that could skew the results, particularly when making high-stakes decisions. At the very end--literally, the second-to-last sentence--the authors say:

"In the end...it is the job of policymakers and educators to define their inferential goals and to decide what kinds of uncertainty are acceptable and what kinds are not."

In other words, understand the imperfections of the data you're using, and make smart judgments accordingly. However, earlier in the same chapter, they say:

"The research base is currently insufficient for us to recommend the use of [value-added methods] for high-stakes decisions."

It is hugely important to understand that they are not answering a research question here. They're answering a policy question, which, as they rightly note in the previous quote, is fundamentally different. No researcher, from RAND or elsewhere, can apply an established methodology to definitively identify the threshold level of error, bias, or uncertainty beyond which data shouldn't be used for policymmaking (other than to make the fairly obvious point that a finding of no teachers effects, or completely flawed methodology, would indicate this. That's not what they found.).

As they rightly state in the previous quote, the degree of acceptable uncertainty--some of which is unavoidable--is a question for policymakers to answer. Moreover, their opinion on this issue is based in part on the lack of research around value-added methods that provides evidence one way or the other, not, as Leo implies, the presence of research that discredits value-added methods.

So I read this sentence to say, "If we were policymakers making high-stakes decisions, we wouldn't use this data." They're not policymakers, but okay.

However, in the summary of the book, at the front (that is, the only part non-statisticians are likely to read), this sentence has been edited down to the following:

"The research base is currently insufficient to support the use of [value-added methods] for high-stake decisions."

Note the omission of the words "for us to recommend." Now it appears that RAND has made an absolute, empirical determination of whether value-added data is good enough for policy. This is the quote that will be frequently brandished and misused if this debate gets off the ground. And it simply doesn't mean what people like Leo will say it means.

Fight for the Right

Apparently, Alexander Russo is from the "big, flashy ideas, however dopey, are superior to potentially promising small-bore ones" school of education analysis. Guess he missed this debate in D.C. last year, where I think similar arguments apply.

Maintenance Matters

This makes me mad. In a country as wealthy as ours, shouldn't teachers be able to expect that, when some basic part of the physical infrastructure of their classroom is broken, it will be fixed so that it works properly, in a timely fashion, and with the minimal possible disruption to the classroom? What other professions put up with this idiocy? Of course, I live in D.C., where 30 schools lacked heat in at least some rooms during the past frigid week, and some school buildings had to be closed several days because of lack of heat. Is it any wonder some people support the Mayor's bid to take over? Education is complicated. Improving low-performing schools is very difficult. By comparison, maintaining facilities should be simple. If our school systems can't even handle such basic functions that tons of property management companies in every major city carry out competently on a daily basis, how the h@#$ do we expect them to meet the much more demanding and complex challenges of raising student achievement?

Bizzaro Idaho

Richard Colvin's been doing a great job covering the through-the-looking-glass-ish debate on preschool in Idaho. Not only is Idaho one of the handful of states that doesn't have a state-funded preschool program;* state laws actually prohibit schools from serving kids under the age of 5. So the state legislature is now debating whether or not to allow school districts to serve 4-year-olds, provided that no state funds are used to do so. I find it particularly wacky that the primary opponents of this measure seem to be conservatives, who are opposing it on grounds that educating kids under 5 in the public schoolsis some kind of assualt on the family (how 5 got to be the magic age when publicly funded schooling no longer infringes on the family is not clearly explained). I thought conservatives believed in local control and flexibility in education? Wouldn't both those ideas imply you should allow school district leaders who believe they could serve kids (and, uh, families) better by offering preschool to do so? Particularly considering that a school district that's gonna bother finding its own source of non-state funding for such services isn't exactly acting on a whim and probably thinks this is really important?

*The federal Head Start program has grantees in Idaho, and presumably the state's public education system is meeting its obligations to serve infants, toddlers and preschoolers with disabilities under IDEA without enrolling them in public schools.

ASPEN on NCLB

The Aspen Commission on NCLB released its long-awaited report today. I'm still reading it, but on first blush it seems like a very solid piece of work, making substantive recommendations in the right places and not getting much wrong. In particular, it goes front and center with the idea of fundamentally changing the way teacher quality is defined, moving the emphasis from qualifications--academic credentials, state certfication, years of experience, etc.--to effectiveness--actual, empirical success in helping students learn. My colleague Eduwonk thinks the report could have done more still to address the human capital challenge in education, and he's right, but this change in and of itself could have far-reaching implications for the basic character and organization of our school system, which I described in a paper I wrote for the Education Trust a few years ago.

Putting the teacher quality recommendations first also send an important message: the standards and accountabilty movement of the last two decades has gotten us to the point where we have a pretty decent idea of what students need to learn, and which schools and districts aren't successful in meeting that goal. The much harder challenge going forward is figuring out what do with those schools and how to help those students, and any successful strategy in that regard has to begin with teachers.

More on this sure to come in the next few days.

Obama on Education

Interviewed on 60 Minutes:

OBAMA: When we're having education debates here in Washington, my positions are informed by having tried to figure out how to fundamentally change the way that we finance public education at the state level. It's informed by work that I've done as a community organizer in inner city classrooms. And so I end up recognizing that we need more money to fix our schools, but we also need a transformation in attitudes. And in Washington, that's typically framed as a "either/or" proposition. You know, the conservative position is we don't need more money; we just need to blow up the bureaucracy. You know, on the left, sometimes the sense is we just need more money, and we and our problems will be solved. When you have actually been in these schools and worked with these parents and talked to the teachers and sat down in a meeting with principals who are trying to figure out how to hold this thing together, then you realize that it's not an "either/or" proposition. It's both ends. You know, parents need to do a better job of parenting. Teachers need to do a better job teaching. Some of the anti-intellectualism that exists in the African-American community and Latino communities and low-income communities has to change. And the federal government's got to put more money, because the fact is that they don't have enough resources.

As an education person, I'm psyched to hear a candidate voluntarily deciding to talk about education prominently during an interview. And I really don't have anything negative to say about this. Sure, it sounds a bit like this, but instead of making it sound like a reductionist political calculation, Obama makes it sound drawn from real experiences with kids, families and schools, like a proper politician and good public speaker should. And he gets a nice Bill Cosby-ish hit in there, too. Certainly, I hope whatever debate over education there is in this election resembles the tone of his remarks more than these.

The Wire Once More

The debut of the fourth season of "The Wire" in the U.K. gives me a chance to go on about how great it is one more time, via the Guardian Unlimited.

Teachers SASS back

Interesting new data from the federal government’s School and Staffing Survey (SASS) released last week provides an interesting look into teacher attrition and mobility. The data comes from a follow-up survey of teachers in 04-05 based on the sample included in the 03-04 data collection. The great thing about this data is that it includes teachers who changed schools or left teaching—a subset of people that are often hard to track down for research. The study is also valuable because its sample includes nearly 7,500 teachers, of whom almost 92 percent responded to the survey.

Some highlights that I found interesting:

• The percent of teachers leaving the classroom has steadily increased from 5.6% in 88-89 to 8.4% in 04-05 (somewhat due to an aging workforce).

• Among teachers with no full time teaching experience, nearly 20% departed in 04-05, more than double the rate of any other group.

• Teachers in high minority schools are the most likely to move or leave.

• Teachers moved to other schools for a better teaching assignment, administrative support, and workplace conditions.

• Teachers left K-12 schools to retire (30%), work in education outside of teaching (29%), care for family (13%), and work outside education (12%).

The part that I found most interesting was the perceptions of former teachers now working outside of education. Compared to their current jobs, the only aspects on which teaching compared favorably were in benefits and the opportunity to make a difference in others’ lives. On every other measure, respondents felt their current position was superior or similar. On several dimensions, these differences were huge—65% cited a more manageable workload in their current work, 65% felt better able to balance personal life and work outside of teaching, 64% felt more autonomy or control over their own work, and 61% noted better general work conditions. These responses dwarfed the more common things you might think teachers were seeking improvements in: salary, prestige, performance evaluation, safety, etc.

It’s important to take these with a grain (or lump) of salt. These are the people who were unhappy enough to leave teaching, and it’s possible that some of them may have been less effective teachers—unfortunately we have no way to know. Also, we don’t know the types of positions that they have moved into—only that on average they did not experience dramatic salary changes.

However, these findings do suggest that there is a fairly significant problem of teacher support—whether that comes in the form of mentoring, planning time, or other measures to help teachers manage their workload and find balance. While I don’t often agree with Linda Darling Hammond, her description of most teachers’ early careers as “hazing” does ring true to me.

In addition to supporting teachers, this data suggests that working conditions and autonomy in teaching lag behind those in other professions. While complete autonomy is impossible in a standards-based environment, allowing successful teachers to innovate seems like a clear win-win to me. And finally, not to ride a union hobby horse, but the poor working conditions in many schools are totally inexcusable both for students and teachers. Schools not having heat (including Simon ES where I taught) is only the latest shameful example in D.C.