Friday, January 25, 2008

This is Probably True

Per The Onion:

TV Critics Admit To Never Having Watched The Wire

NEW YORK—Despite heaping lavish praise on the HBO crime drama The Wire, television critics across the country admitted Monday that not one of them has ever sat down to watch an entire episode of the show. "The Wire has done what no other television program has come close to achieving—namely, presenting the life of a decaying American city and doing so with the scope and moral vision of great literature," said New York Times critic Virginia Heffernan, who was surprised to hear that the groundbreaking series had already started its fifth and final season in early January. "It sounds fantastic. I really wish I had HBO." Many reviewers from top media outlets assured reporters that they would start watching the Peabody Award–winning show just as soon as the first season reaches the top of their Netflix queues.

"Technical" Objections

Over at the UFT, Leo Casey accuses us of various rhetorical sins involves caricatures, straw men, etc, in recent comments about the NYC value-added project. Their grievances lie with the methodology, says Leo, and it's wrong to say otherwise. Okay, very well. Two questions:

1) What are the basic elements of a UFT-approved methodologically appropriate method for estimating individual teacher effectiveness using value-added measures based on standardized test scores? For example, if the school system were to find a way to fix every problem that Leo mentions--multiple teachers between tests, small numbers of students per teacher, etc.--would that be sufficient to assuage their concerns?

2) Given a UFT-approved value-added methodology, what uses of value-added data would the UFT endorse? Could the results be released to the public? Could they be used as one factor in making tenure decisions or providing performance bonuses? Anything else?

If Leo provides a straight answer to those questions, and if the answer to the second questions is something other than "None," I'll officially apologize for saying that the UFT's reaction to the value-added reflects a principled opposition to evaluating teachers using student test scores.

Thursday, January 24, 2008

No Child Tickets

En route to a meeting here in D.C., I walked by the Woolly Mammoth theater this morning and saw this poster for the solo show by former NYC public school teacher, Nilaja Sun. Sounds good, and more fun than most No Child events these days. It opened on Jan 21 and will be around through February 17th.

Massive Endowments















The annual college endowment report from the National Association of College and University Business Officers (NACUBO) was released yesterday. Overall, it was a great year for higher education, with average earnings of 17.2 %. The richest institutions (over $1 billion in assets) did even better, earning 21.3%. One consequence of the growth is that institutions are having a hard time figuring out how to spend all the new money; endowment spending as a percent of assets dropped to 4.6%, the lowest rate since 1999. This will probably provide fresh ammunition to those in Congress and elsewhere who have proposed that institutions be required to spend a fixed percent of assets--usually 5%, the legal standard applied to many non-profits already. As Rich Kahlenberg said in the Wall Street Journal, "The price of college is rising and endowments are growing and people are frustrated by those two things going on at once."

On some level, the top schools are victims of their own success. As has already been widely reported, Harvard earned $5.7 billion on its endowment last year, which is by itself larger than the total endowment of all but a handful of institutions. This is the principle of compound interest in action--when you meet with a retirement advisor, they always show you the parabolic curve of expected earnings and point out that you make most of your money in the last 10 years. Universities have a longer investment time horizon than any other institution that currently exists, governments included, so it's no suprise to see their longevity paying off.

But all that money raises some uncomfortable questions, about the price of college as Kahlenberg suggests (it's no coincidence that Harvard and Yale timed the announcement of their new endowment-funded financial aid programs to hit the news in the weeks before the NACUBO report), and--more importantly, in my opinion--why, exactly, all of this is being subsidized by the government in the form of tax preferences for everyone involved. As Richard Vedder pointed out in the Post over the weekend, Princeton recently built a new residence facility, Whitman College (above), named after major donor and alumna Meg Whitman, CEO of Ebay, which cost a staggering $388,571 per unit, roughly what Donald Trump spends building a luxury resort. Here we have a fabulously wealthy person donating money to a fabously wealthy university to built a fabulously expensive facility for the benefit of students who come from, in many cases, very wealthy families. I have no problem with that personally if that's how they want to spend their money, but why am I, as a taxpayer, footing part of the bill?

Wednesday, January 23, 2008

Value-Added Comes of Age

About four and a half years ago, I was working on a policy paper focused on a developing and controversial method of measuring teacher effectiveness called "value-added." Created by Dr. Bill Sanders in Tennessee in the mid-1990s, the essence of value-added is pretty simple: Using annual standardized test scores, look at the prior achievement history of a given teacher's students and, based on that, statistically predict how well they're likely to do in the current year. Then calculate the ratio of their actual performance to the predicted performance. Teachers with a ratio greater than 1.0 are more effective than average, those with a ratio of less than 1.0, less. This gives teachers credit for making a lot of progress with previously underperforming students, and doesn't give them progress for coasting with previously high-performing students. Whatever external factors impact performance--poverty, family life, etc.--are implicitly controlled for, because the prediction model is based on the prior performance of the students themselves.

Around the same time, I was reading Michael Lewis's new book, Moneyball, which I had bought the day it was published, being a big fan of both the author and the Bill Jamesian approach to thinking about baseball. And at some point I realized that the underlying premise of Moneyball and the promise of value-added were the same: using empirical data to fundamentally change and improve a labor market. Instead of relying on human observations of characteristics, with all the biases and errors that result, focus on outcomes instead. The paper was released the following year, my first and only real contribution to the teacher quality debate, and while in retrospect I'm kind of embarrassed by the length, I think the ideas hold up pretty well. (The original draft included a whole section explicitly drawing the Moneyball parallel, but it was excised in the editing process, and yes, I'm still bitter.)

So it's interesting to see (I take no credit for this) the New York City school system announcing a plan to start calculating value-added scores for some of its teachers. Just like in Tennessee, the idea is pretty straightforward:


The city’s pilot program uses a statistical analysis to measure students’ previous-year test scores, their numbers of absences and whether they receive special education services or free lunch, as well as class size, among other factors. Based on all those factors, that analysis then sets a “predicted gain” for a teacher’s class, which is measured against students’ actual gains to determine how much a teacher has contributed to students’ growth.

What they're going to do with the data, however, is unclear. One option is to use it for making tenure decisions, which, research has shown pretty conclusively, is a good idea--it seems clear that if your value-added scores put you among the very worst teachers in your first few years--like the bottom 3%, say--the odds of you ever becoming a good teacher are quite low. As Harvard's Tom Kane notes, "It seems hard to know who is going to be effective in the classroom until they are actually in the classroom.

Or you could simply put the data out there and let market forces work. Deputy School Chancellor Chris Cerf said:


“If the only thing we do is make this data available to every person in the city — every teacher, every parent, every principal, and say do with it what you will — that will have been a powerful step forward. If you know as a parent what’s the deal, I think that whole aspect will change behavior.”

Crucially, this would be good for the best teachers. One of the biggest problems with the teacher labor market is that the top teachers--the ones who are one or more standard deviations above the mean in terms of effectiveness--are criminally underpaid, and have no way of demonstrating their real value to the labor market. Their unions, however, are totally aghast at the prospect. Randi Weingarten, head of the United Federation of Teachers (and rumoured to be next head the national AFT) said:


“Any real educator can know within five minutes of walking into a classroom if a teacher is effective."

This is the equivalent of the scouts and general managers in Moneyball who were always on the lookout for the "good body," the "five-tool guy," the player who just looked like a major leaguer. As everyone now knows, they were profoundly mistaken, and people like the Oakland A's Billy Beane were able to exploit the market distortions that resulted.

What we're seeing in New York City today is all the major challenges of 21st century K-12 teacher policy being played out in real time. Value-added methods are still very much in development, subject to limitations of standardized tests, among many things. But in the long run, there will only be more, better information about student performance, along with newer, faster ways of analyzing that information and drawing increasingly accurate conclusions about how well teachers are doing their jobs. At some point the methodological debates will be resolved and the margins of error whittled down the satisfaction of reasonable people.

That will have profound implications for the way teachers are hired, paid, trained, assigned--perhaps for the nature of the profession itself. Much current teacher policy is logically derivative of extremely limited or absent information--if we can't accurately measure teacher effectiveness, then pay everyone the same. If we can't know how well teachers will perform when they arrive in the classroom, throw up a lot of regulatory and process barriers to entry in terms of training and certification. The shift from information scarcity to abundance will change that logic, and eventually the policies themselves. New York City is a sign of things to come.

Update: Eduwonkette compares this to the infamous Tuskegee syphilis experiment, and then says she's not actually making the comparison she just made. The privileges of anonymity, I suppose. Sherman Dorn chooses a different horrible disease (botulism) to make his point--which is that the NYC value-added process may or may not have severe methodological flaws. It might, I don't know, I guess we'll find out. But, per above, methodological issues can be worked out, and anyone who thinks the hysterical reaction to the value-added initiative stems from a deep and abiding concern for statistical integrity is willfully not paying attention.

Update II: Dorn updates and points out that the "botulism" reference wasn't hysterical, fair enough, I was referring to the folks at UFT but that wasn't clear. He also says:

The claim that "methodological issues can be worked out" is evidence that Carey hasn't read the writings of professional researchers who point out that growth models are no holy grail.

I've read the research (which the UFT habitually misrepresents) pretty carefully. The people who've looked at the Sanders model have generally concluded that it does what it says it does: identify teacher effects, given appropriate caveats about statistical margins of error. It's true that they say it's no holy grail, which is unsurprising in that there's no such thing as a holy grail. There is not now nor will there ever be perfect information about teacher effectiveness; teaching is far too complicated for that. The only responsible approach to using value-added data--or any other data that purports to gauge teacher effectiveness--is to be cognizant of the amount of likely error and craft policies accordingly. And of course there should also be diligant work to improve the methods themselves, which nobody believes have reached an apex of refinement.

Given that, a fair response to the NYC announcement would be something like this: "We support efforts to fairly evaluate teacher effectiveness and recognize that objective evidence of student learning growth must play an important role in that process. We emphasize that the results of evaluation methods based on standardized test scores will be subject to significant degrees of statistical error, which much be appropriately taken into account, particularly when such information is used in the context of employment matters such as tenure and compensation. The best process will combine information from multiple methods, including peer and principal evaluation, and will preserve teachers' professional rights. We look forward to working in concert with management to develop such policies, which should include rewarding the most effective teachers for the vital work they do."

The actual response from the UFT was nothing like that. Rather, it reflects a principled opposition to the use of test scores of any kind in evaluating teachers. Again, this is not an argument about methodology; it goes much deeper than that. Talking about methods in a good faith attempt to reach the goal of better information is one thing, but the holy grail standard is all about making perfection the enemy of the good.

Tuesday, January 22, 2008

Wisconsin, Cyberschools, and Virtual Schooling

The publicity following a recent ruling by the Wisconsin Court of Appeals ordering the state to end funding of the Wisconsin Virtual Academy, a full-time virtual charter school operated by the Northern Ozaukee School District (just north of Milwaukee), is helping to reinforce the public misconceptions around virtual schooling. Take, for example, this breathless lede in Friday's AP story:


School districts across the United States are watching a court ruling that challenges the existence of virtual schools and could determine the future of online education.
Don't believe the hype and don't think that all virtual schooling looks like Wisconsin Virtual Academy. Many people think that "cyber" charter schools, schools that are responsible for students' entire education experience and that students attend full-time, are the primary sponsors of online learning. But, the majority of students learning online participate in "supplemental" virtual schooling programs. These supplemental programs, many state-run, allow students to take online courses in addition to their regular school-based courses.

The AP story and much of the discourse surrounding the Wisconsin decision confound the ideological and political controversies that surround cyberschools with virtual schooling in general. Despite their growth and popularity, full-time cyberschools are highly controversial—a result of their non-traditional approach to learning, their status as charter schools, the transfer of student funding away from traditional schools, and their enrollment of former home-school students. Many unions, especially state and local- level affiliates, have vehemently opposed cyberschools.

Not only is the story above factually incorrect (the law in question is a very specific Wisconsin state law—Colorado, for example, just recently passed its own cyberschool legislation), but it narrows all of online learning to the cyberschool model. This is particularly dangerous because many of the practices found in supplemental virtual schooling programs are bringing about reforms that have long eluded traditional public schools and prompting educators and policymakers to question and change key components of our traditional, classroom-based public system. Limiting the discussion of virtual schooling solely to cyberschools equates virtual schooling with home schooling and bounds the possibilities of online education into the constraints of polarized, non-productive and ideological education battles.

That said, proactive state legislators across the country should take note of the Wisconsin case. Many states' laws, like those in Wisconsin, need to be modernized to reflect an education system that is no longer defined by bricks and mortar schools, seat time, and strict geographic lines. These legislators should heed the lessons of the charter school movement and craft strong laws that provide the regulatory and accountability framework needed for all forms of virtual schooling within the public system.

Specifically, cyberschool options should exist to help serve the vast variety of student and family needs. It's ok for different models to serve different students with differing needs to ensure each of them is successful—that's called personalization.

The key issue is really around public funding. Funding should follow student success and not penalize innovation or efficiency. But, let's also make sure to weight the public funding so that it is sensitive to what the state is actually buying. Many cyberschool models have very limited student/teacher interaction, rely heavily on parents as educators, and are less likely to serve special needs students or those without a stay-at-home parent. With weighted funding, some cyberschools would receive less money. But, more money could also follow to cyberschools that can demonstrate success serving students with higher needs. The more sophisticated conversation is not about whether cyberschools are good or bad, but what role they play in a state's various educational priorities and how the funding follows those priorities.

Funding Gaps

The Education Trust released its annual state report, The Funding Gap, which looks at funding trends from 1999 to 2005 and compares district funding for those serving the highest percentages of poor students and students of color and those serving the lowest percentage of these students. Winners include 10 states that narrowed the gap between funding for low and high-poverty districts: Alaska, Arkansas, Connecticut, Minnesota, NJ, NY, New Mexico, Ohio, Wyoming and my own Maryland. Maryland, btw, along with Ohio and Wyoming, not only closed their gap but reversed it and now provides more funding to high-poverty districts. Losers? Illinois, Florida, Texas and more than a dozen more states where the funding gap actually increased from 1999 to 2005. That's right smack in the middle of our strong accountability era, where all school systems are pressed to meet standards for all students. How can we have school financing policies that are so in conflict with this accountability standard? It seems to me that aligning our accountability and finance systems should be a basic must-do.

Also good to note that EdTrust has added an important new piece of analysis to their annual report. Carmen Arroyo, the author of the report, crunched the numbers on percentages of English Language Learner students and found that in the eight states with the highest percentages of ELLs, districts with a lot of ELLs receive less funding than districts with few or no ELL students (I'll have to ask Carmen how many districts in the top ELL states really have no ELL students? Few, I'll bet). This type of ELL analysis is a key addition, as this population of students promises to grow not only in those high-ELL states but in most others too.

Monday, January 21, 2008

The Wire, Season Five, Episode 3

Summary: McNulty successfully fakes evidence of a Baltimore serial killer, yet nobody cares. Lester explains that he's failed to include the kind of lurid psycho-sexual element that will bump the story onto the front page. Bunk is increasingly (and amusingly) aghast. Prop Joe gives Marlo money-laundering lessons. The Sun's parent company hands down a new round of budget cuts, forcing the closure of all foreign bureaus and firing / buyout of veteran reporters. The Fabulist makes up some more stuff while acting like even more of a snot. Burrell tries to feed Carcetti fake crime stats, not realizing that Valchek gave the mayor the real stats first, so Carcetti sets the wheels in motion to can Burrell and replace him with Daniels. Clay Davis continues to feel the heat. I think I could watch a whole hour of nothing but Clay Davis. Michael begins to conclude that certain aspects of working for the Marlo organization--such as working seven days a week on the corner and being asked to murder small children--aren't so great. Method Man gives up Butchie to Chris and Snoop, who promptly torture him (Butchie) to death in an effort to smoke out Omar. He was indeed a tough old man. Snoop continues to question the wisdom of the "provoke the baddest man in Baltimore into trying to kill us all" strategy. Omar, blissfully on vacation in the Carribean with his boyfriend, hears the news about Butchie. Duh-duh-DUH!!!!

This episode continues the proud Wire tradition of constructing scenes that assume you've studied every scene in every episode with Talmudic intensity, like when Daniels and his ex-wife fret about Burrell possibly revealing old allegations of Daniels doing something vaguely corrupt back when he was involved in drug investigations, before the show even began. I can't remember the last time this fact was even mentioned, and its never been clear to me whether the allegations are even true, but nothing is ever forgotten on The Wire, because it's all connected. Bonus: Pearlman grilling Clay Davis' driver about the $20,000 he got caught with all the way back in Season One.

Now that Omar has reappeared, the most signicant characters yet to show up are, in no particular order:

Cutty
Prez
Bunny
Namond
Randy
Poot
Royce
Brother Mouzone

Episode three seems like as good a time as any to put The Prop Joe Question on the table. To wit: the "prequel" webisode featuring Joe is dated 1962, where he looks to be about ten, give or take. That means Joe is now roughly 55 years old. Allowing for time to work up through the ranks, etc., he's been seriously in The Game for 30 years, minimum. Joe, more than anyone else on the show, has figure out the two essential truths of The Game, and their resulting corollary: 1) Making money is the easy part. 2) Staying alive and out of prison is the hard part. Ergo, the smart play is: Whenever possible, trade money for a reduced risk of being imprisoned or killed. But even given the resulting reduced income, Joe must be fabulously rich; heroin has been in Baltimore since the 70s and crack since the 80s. As we learned this week, to nobody's surprise, Joe knows how to launder money into offshore accounts. His position as the middleman betweent The Greek and the rest of the Co-op members must be extremely lucrative, I assume he's the equivalent of "The Bank," the position Stringer Bell was shooting for before he got shot. Yet Joe must also know that while he can reduce the odds against him, he can't ever eliminate them entirely. Eventually he'll roll snake eyes and get caught in the Marlo-Omar crossfire, or something along those lines. You can have a short run, or you can have a long run, but only The Game remains.

Given all that, what the heck is Proposition Joe doing in East Baltimore? Why does he spend all day in the back of a repair shop in his ratty hoody, dressed liked Kevin Smith, while bad men with guns come and go? He's a smart guy -- why not learn French and retire to Antigua? I don't get it. I suppose the only plausible explanation is that he just enjoys The Game for its own sake. Sam Zell is pushing 70 and has $6 billion, yet he's busy buying media companies, screwing around with Wrigley Field, and imposing real-life cost cuts on newspapers like the real-life Baltimore Sun. Same principle, I guess. The only other explanation is some kind of horrible last season retcon involving numerous heretofore unmentioned losing trips to Atlantic City and Vegas, a la The Sopranos, but I think David Simon is too smart for that.

A Faulty Argument

Back in December, I blogged about a proposed Higher Education Act provision that would change the way default rates are calculated by the Department of Education. I hypothesized that this change could increase default rates by about 60 percent, putting some schools at risk of losing their eligibility to participate in the federal loan program because of too-high default rates -- if a school has a default rate of 25 percent or higher over three years or a single-year default rate of 40 percent or higher, it loses eligibility to participate in the federal loan program.

The proposed change is pretty simple. Currently, the Department of Education calculates the student loan default rate as the percent of students defaulting on their student loans in the first two years after entering repayment. The proposed change would extend that window to the first three years after entering repayment. But, as we showed in an October Charts You Can Trust, the percentage of students defaulting on their loans grows considerably with each added year, particularly among students with high amounts of debt.

And now, as Inside Higher Ed reports, the Department of Education released numbers showing that for-profit colleges would see their default rate double on average, and other institutions could see their rates increase by between 50 and 75 percent (check out the article for numbers broken down by institution type). Not surprisingly, for-profit colleges are not happy about this change and are trying their best to resist it.

The Career College Association is quoted as saying, “This is a questionable policy metric because community colleges, proprietary schools and minority serving institutions all accept a much higher percentage of lower income students than do traditional schools, and many have a higher [Cohort Default Rate] as a result. The single best predictor of a student’s likelihood of default is the student’s own socioeconomic status.”

Sure, socioeconomic status impacts a student’s risk of defaulting on loans and students should definitely be held responsible for responsibly managing their debt. But colleges have a responsibility to equip students with the skills and knowledge needed to get the jobs that allow them to repay their debt, and for ensuring students graduate on time -- or at all. And schools are also responsible for ethically counseling students on the amount of debt they can realistically undertake for their education and for providing adequate financial aid so that students are not graduating with overly burdensome debt loads. All of these actions are entirely within a college’s purview and would have a large impact on the percent of students defaulting on their loans.

Also, take a look at the Department of Education cut-off default rate numbers: 25 percent default rate over three years or a 40 percent default rate over one year. Is a college that consistently allows one in four of its students default on their loans in the first three years (in reality, two years*) after leaving college really providing a valuable service that the student and taxpayers should be paying for?


* Because it takes nearly a year after a student stops paying on his or her loan for the loan to technically be considered in default, a three-year window is really a two-year window, and a two-year window is really a one-year window. In addition, students with loans in deferment or forbearance are counted as being in repayment.