Commendable Commentary

This blog hasn't had comments for that long, and I'm hoping more people will use them as a chance for substantive dialogue that focuses on the topic of the post in question, as opposed to anonymous shot-taking and/or general complaint. In that spirit, let me heartily commend the second comment in this post, in which the author identifies herself by name and offers some intriguing new information and perspectives about inter-district choice and the example of Piedmont Unified School District in Oakland. I had written:

The starred school in middle is located in Piedmont School District, which is very affluent and literally built on a hill surrounded on all sides by Oakland Unified School District, a high-poverty district with rock bottom test scores that's been subject to state takeover in the past. There are four low-performing Oakland schools within a five minute drive of the Piedmont school and at least nine within 20 minutes. I think we can all agree that even under the rosiest of scenarios, the Piedmont school couldn't quintuple in size, or more, to accomodate transfer students.

To which she responded:

Hi: I live in Piedmont, which is now surrounded by very affluent areas of Oakland (including, I believe, the four top-rated elementary schools in Oakland, which sometimes SURPASS Piedmont's elementary schools in their API scores--Thornhill, Montclair, Joaquin Miller, and Crocker Highlands).

When Piedmont was chartered as a city 100 years ago, it was a dairy settlement way up above the city of Oakland, which at the the time was centered right next to the Bay. Over the past 100 years, Oakland has grown up and around pre-existing Piedmont. Home prices in Piedmont average about $300,000 more than similar houses across the line in Oakland. About $1.3 million compared to $1 million, for a typical 3/2 home in the three Piedmont zips and the corresponding Oakland zips (that is, Oakland and Piedmont are both in 94611, 94610, and 94618).

"Parcel taxes" for schools (because, of course, property taxes are strictly limited across the state by Prop 13) are now at about $1500/home above what a home in Oakland would pay (zero). Realistically, the PUSD would never open its doors in a serious fashion to Oakland residents, because the value of property across the town (3400 homes x $300K each = about a billion dollars) would drop by about a billion dollars.

But note that the big beneficiaries would NOT be high poverty communities, because surrounding areas are not high poverty. They are high-income--Montclair, Claremont Hills, Claremont Pines, Crocker Highlands, Ridgecrest, Piedmont Pines, Piedmont Avenue, etc.

Interesting! A couple of thoughts on this. As Erin notes below, some have critiqued the framing of our report, which estimates that "only" 10 - 20 % of students could benefit from inter-district choice, when that's in fact a lot of students and thus the findings could just as well be presented in more positive terms. Which is a fair point; realistically, improvement happens through the accumulation of multiple initiatives each doing their part.

But it's important to keep in mind that those numbers are a best case scenario given the assumptions we chose. That's the number you get if every single higher performing school opens its doors to out-of-district students and the maximum possible number of students choose to travel. As the comment notes, there are, in some communities, very powerful social and financial incentives for high-performing districts not to make that choice. That doesn't mean those barriers can't be overcome, but it's safe to say that commensurately large amounts of political and financial capital will be required to do so.

And that raises the issue, per Erin, of balancing costs and benefits. You can bribe high-performing districts to accept transfer students, but that by definition involves directing scarce resources to schools that likely have the fewest educational needs. Inter-district choice also creates signficant new transportation costs, born either by parents or the taxpayers, directing resources to activities that are fundamentally non-educational in nature. Connecticut spends something like $3,000 per student to bus Hartford kids out to the suburbs. That's about what some of the best urban charter schools spend above and beyond normal per-student allocations to extend the school day, provide extra tutoring, lower class size etc.

Inter-district choice can be a good policy for some students in the right circumstances. But it's not going to absolve us from the pressing need to build more, better schools, including more schools of choice, in the neighborhoods and communities where disadvantaged students actually live.

Technical Note: As I should have made clear in the original post, all of the schools on the Piedmont map are high schools. It's true, as the commenter notes, that there are some high-performing elementary schools in Oakland near the Piedmont border, and they're in an attendance area that feeds into a high-performing Oakland high school (the other star on the map). But there are still a whole lot more low-performing high schools in Oakland than high-performing ones, and the students there can't all transfer to Piedmont.

Say No to Cheap Choice

On Tuesday, Matthew Yglesias commented on the report ES released this week on interdistrict school choice (see below for more posts about this). Yglesias questions my use of the word “only” to discuss our finding that 10 to 20 percent of students would likely benefit from interdistrict choice. Instead he argues that for the students who do benefit, it’s worthwhile. This brings up an excellent question – if a reform is likely to impact a minority of students, but will substantially benefit those students, is it worth the costs? Or, more appropriately, how much in additional resources is it worth expending for the benefit of a minority of students?

This is a question that plagues many choice proposals, which often can only serve a minority of students. And with interdistrict choice, the costs can be pretty high, so policymakers would rightfully want to impact a substantial number of students, even if it isn't the majority. Effective interdistrict choice requires additional transportation to get students to their new schools, financial incentives to get higher performing schools to participate, and academic support for transferring students. It’s critical that a school district or state interested in implementing interdistrict choice carefully weigh these costs with a realistic estimate of the number of students who will benefit—in some areas it will be a lot, and in others only a few. A worse-case scenario is a policymaker trying to scrimp on these costs— introducing a “cheap choice” program that is ultimately detrimental both to goals of integration and student achievement.

Reality Time

Matt Yglesias observes the spate of union-sponsored Fenty-bashing at the Democratic national convention, driven in large part by the Fenty administration's proposal to pay teachers a lot more money in exchange for more accountability and less job security, and doesn't like what he sees. (Side note: let's all agree to apply some standards of objectivity and reasonableness in the use of the word "bashing." If you're, say, handing out flyers that call someone a "budget-shattering, union-busting, promise-breaking political boss whose poor performance and bad management are costing DC taxpayers millions of dollars," that's bashing. If you say "Fenty's policies would be bad for teachers and public education, and here's why," or, alternatively, "The union position is short-sighted and will degrade the quality of the teacher workforce," that's not bashing, which implies a certain level of name-calling, histrionics, and barely-concealed rage.) Matt notes that:

DC is, at this point, in better financial shape than the vast majority of American localities and also has much worse schools. Under the circumstances, it’s the best possible opportunity for teachers to get what Rhee’s putting on the table — generous reform that puts real resources on the table and thereby keeps teaching as an attractive career path even while building some additional accountability into the deal.

Teacher salaries have been stagnant for a long time. They don't get paid as much as other other well-educated professionals whose jobs require similar levels of training and hard work to do well. Teachers and their representatives in organized labor think this is unfair and would like it to change. And they're absolutely right to do so. It would be great to see teachers consistently making six-figure incomes. A lot of them deserve it, and it would send powerful signals to the job market about the nature and status of the profession.

But the idea that those kinds of dollars are going to arrive without some significant tradeoffs in terms of accountability and job security is a complete fantasy. It will never happen. And the circumstances under which it can happen are fairly uncommon: A powerful sense of need--i.e. a school district whose educational challenges justify the infusion of resources--plus leaders with access to those resources and the willingness to let teachers decide for themselves whether or not to participate. In other words, the deal isn't getting much better than this.

Critics have raised the usual objections about the potential for favoritism and bias in the evaluation process, the spectre of good teachers losing their jobs because they happened to get a particularly tough group of students one year, things went pear-shaped on testing day, etc. But let's think about that for a moment. Here you have a mayor and schools chancellor who've staked their careers and reputations on turning around the school system. To do that, it's vitally important for them to hire, retain, and support good teachers. It would, therefore, be shockingly dumb to start arbitrarily firing good teachers or otherwise treat them unfairly. Fenty and Rhee have every incentive to make sure their evaluation process is sound. If they don't, their jobs are the ones at risk.

This highlights the importance of understanding how the major threads of school reform fit together. Absent political or governmental accountability for results, it's perfectly reasonable to worry that management at various levels--city, district, school--could and will abuse their discretion. If nobody cares whether your students are learning, then sure, give a fat bonus to your buddy and fire the teacher down the hall who called you out for your incompetence. But when there's real accountability and public scrutiny, the incentives change, and policies like tying teacher pay to performance, defined in part by managerial judgment, start to make a lot more sense.

Space, Time, and Inter-District Choice

Just to amplify a few points Erin makes below about Education Sector's new report examining inter-district school choice: One of our main goals in conducting this analysis was to try to get a handle on how the generalized--and very worthwhile--idea of expanding school choice across school district boundaries plays out given real world contraints like geography, transportation infrastructure, and the uneven distribution of high-performing schools.

For example, in an upcoming report Erin will be running the same Geographic Information System (GIS) analysis for the city of Chicago. It will show that only a limited number of students there could benefit from the opportunity to transfer to other districts, particularly in high school. This is partly because of issues like capacity and distance. But it's also because Chicago was built next to a gigantic lake that contains no schools, good or otherwise. So while Chicago students might be able to travel north, south, or west to find better schools, they pretty much can't go east. These things matter.

The way school districts are designed also makes a big difference. Some states--Illinois is one--have hundreds of small districts. Marion County, Indiana, where I used to live, contains eleven separate school districts, including the traditionally urban Indianapolis public schools along with various large and small township school districts, including the tiny district of Speedway, which enjoys unusual wealth owing to its ability to levy property taxes on the Indianapolis Motor Speedway, where the Indy 500 is contested. Inter-district choice in Marion County would undoubtedly benefit many students.

On the other hand, many southern states operate large, whole-county school districts. The map below shows a low-performing school that happens to be located in the middle of the Polk County, Florida district, which encompasses over 2,000 square miles. It takes 45 minutes just to get to the county border from that school, much less find a better school on the other side. Inter-district choice isn't really an option there.



















Some critics of the report have raised the issue of our school capacity assumptions. It's true, as Dianne Piche notes below, that some schools may have more capacity than they'd like to admit. But others have suggested that, absent any definitive empirical evidence of how much successful schools could theoretically expand to accomodate new students, we should have either (A) made no assumptions at all, which is methodologically the same as assuming that schools have infinite capacity to expand, or (B) treated all possible assumptions as equally valid. Frankly, that doesn't make much sense, and the map below shows why.




















The starred school in middle is located in Piedmont School District, which is very affluent and literally built on a hill surrounded on all sides by Oakland Unified School District, a high-poverty district with rock bottom test scores that's been subject to state takeover in the past. There are four low-performing Oakland schools within a five minute drive of the Piedmont school and at least nine within 20 minutes. I think we can all agree that even under the rosiest of scenarios, the Piedmont school couldn't quintuple in size, or more, to accomodate transfer students. There are limits, and any responsible analysis of inter-district choice has to acknowledge that. Piedmont is, of course, somewhat of an extreme example, but that's why we ran the numbers for every single school in California (and Texas and Florida) using a moderate capacity assumption, to see how things play out in the aggregate.

Assumptions

In Dianne’s response below, she talks about two important assumptions we've made in ES's recent report on interdistrict choice--one assumption about driving distance and the other about school capacity. Richard Kahlenberg, an ES Senior Fellow and a well-known advocate of using interdistrict choice as a means to achieve economic integration also takes issue with the assumptions we made regarding these two variables, saying that we’ve artificially limited the outcomes by choosing a driving distance and a school capacity estimate that are too modest.

I disagree with Kahlenberg that the assumptions made in the report are somehow irresponsible, but I do agree that they’re debatable and indicate areas where, as Dianne points out, additional research would substantially further the discussion on the feasibility of interdistrict choice. (Click here to see a full explanation of our assumptions and their impact on the report’s results—also available on page 4 of the report in a full-page sidebar, in smaller, but not fine print).

First up, the driving time assumption. We calculated the potential of interdistrict choice based on the availability of higher performing schools located within a 20-minute driving radius of each lower performing school. It’s important to note that the actual travel time is likely longer because the estimate doesn’t account for traffic or for picking up students along a bus route.

But, even so, is a 20-minute driving time unreasonably short? It might be if you’re a parent desperate for a better school option and willing to transport your kids up to an hour on the bus, but for many parents, most of whom like the idea of their children attending a school that is nearby and easily accessible to them during the day, a 20-minute drive would simply be reasonable. And if interdistrict choice is going to work on a large scale, it needs to work with reasonable commuting distances to schools – distances that allow parents to volunteer, attend PTA meetings, and pick their kids up when needed.

Even if we did assume that parents and students would be willing to commit to longer commutes, our results wouldn’t change by much. This is because, under our model, if one student can travel an hour, then all students can travel an hour. Therefore, increasing the driving distance both increases the available school choices and the competition for those choices. The limits of choice aren’t so much about 20-minutes versus 60-minutes, they’re about how many good schools are out there, how many students are competing to get in, and how much capacity higher performing schools have to enroll students from lower performing schools.

Which brings me to the second assumption – school capacity. Dianne makes the good point that school capacity is a “moving target”. It’s difficult to determine the actual ability of higher performing schools to expand—there is little available data and capacity might change from year-to-year along with population shifts and budgets. The recommendation made by the Aspen Commission on No Child Left Behind to require districts to audit available space in schools would go a long way to removing some of the mystery around school capacity and may, as Dianne predicts, reveal much more space than we’ve assumed is available.

But in the absence of that data, we still needed to make an assumption about the ability of schools to expand to take in transferring students—and we assumed that schools could increase capacity by 10 percent. If we made no assumption about school capacity and instead threw our hands in the air in bewilderment, we would have ignored a hugely important variable in the interdistrict choice equation. By settling on one number as a starting point for discussion, we've been able to move the conversation about interdistrict choice to a place it needed to go—talking about the practical limits and realistic potential of interdistrict choice.

In the end, what I hope readers take away from this report, and what I found particularly interesting, isn’t just the bottom line numbers or debate about our assumptions, but the fact that the potential of interdistrict choice can vary greatly from one community to the next—in some areas, it can have a much bigger impact than the overall numbers might indicate. In East Palo Alto, CA, for example, 35 percent of students could transfer with interdistrict choice--compared with only 9 percent in Los Angeles schools.

And it is this implementation of interdistrict choice programs that is crucial --where will interdistrict programs be most effective and how can they be implemented to best serve the students who are most in need of access to better schools.

Dianne Piché on "Plotting School Choice"

If advocates for children were able to draw school district boundaries anew, nobody in their right mind would configure them with the high levels of inefficiency, inequality and segregation we find today in states like California and Texas (included in the study) and others like New Jersey, Connecticut and Ohio. Education Sector's report on interdistrict school choice underscores the fact that we have consigned large numbers of the nation’s poorest and most academically struggling children--children who are disproportionately African-American and Latino--to schools in districts with the worst track records and least capacity to educate them. Yet, we know that smart, thoughtful approaches to school choice--including providing interdistrict options and successful charter schools, along with parent outreach and incentives for receiving schools--can yield enormous benefits. Low-income parents are empowered and more satisfied. Parents of all races and income levels experience and come to value diversity. Students benefit in tangible and intangible ways. In St. Louis, for example (where I have represented the NAACP and schoolchildren), we negotiated the largest, longest-running inter-district program in the country. Outcomes for the Black students who choose to attend schools in suburban districts are demonstrably better than those who remain in city schools: they have access to more rigorous coursework, and they graduate and attend college at much higher rates.

The good news from this report is the finding by the authors that some 10-20% of eligible students could be accommodated via interdistrict choice under their hypotheses. The sobering news, of course, is that the political will is not quite there yet to help make this happen and that laws like NCLB will need to be amended to strengthen the interdistrict choice provisions.

Finally, while the report provides interesting geographic and demographic data for a handful ot localities, I would argue that its premises, while useful as a starting point, are incomplete in at least three respects:

First, parents who choose demonstrably better schools for their children (whether magnet, private, parochial or interdistrict public) are often willing to have their children transported longer than 20 minutes each way. In the Washington, D.C. area where I live, it is not uncommon for students to commute up to an hour each way to high-performing schools that promise greater access to rigorous coursework and competitive colleges.

Second, in my experience litigating school desegregation cases I have learned that school "capacity" is a nimble & moving target. It is used as a defense by parents and teacher in affluent communities to resist welcoming less advantaged students. Figures rise and fall with school budgets, housing patterns, the commitment of states and the federal government to enforce the law, and, unfortunately, racial dynamics. Recently, in Birmingham, Alabama, the school district argued it could not honor middle school students’ right to transfer out of a school in need of improvement because it lacked capacity in other schools. But when the U.S. Department of Education ordered compliance with the law, capacity was found and the right was extended to all eligible students.

Finally, although the report acknowledges the existence of preexisting choice--i.e., school choice made pursuant to state or local law, magnet programs, informal practices and the like--it does not address the impact on enrollment and capacity. For example, a high school’s enrollment increases and its capacity to accept NCLB transfers is reduced every time an affluent parent chooses to jump district lines and enroll her child in a neighboring district. Because many states and districts allow other forms of choice (with parents usually providing their own transportation and sometimes even making tuition payments to these better schools, in the case of interdistrict choice) and make it available before NCLB choice, there is likely to be more capacity than meets the eye in many schools. Additional research could be helpful in this regard.

--Dianne Piché, Executive Director, Citizens' Commission on Civil Rights

Down and Out of District

That's the title of a Beverly Hills 90210 episode in which Andrea is nearly found out for attending West Beverly High instead of her assigned high school, located in a lower-income part of town. While Andrea gets away with her illegal transfer, many students aren't able to cross district boundaries to attend a better school. In these cases, district boundaries serve as an artificial barrier to choice, often limiting school options to the low-performing schools within district boundaries -- boundaries that often reflect the racial and economic segregation of larger society.

It was this often arbitrary and segregating effect of district boundaries (and not the 90210 episode) that motivated us to look at the potential of interdistrict choice to offer more options to students in low-performing schools. To our surprise, we found that interdistrict choice on a large scale is unlikely to benefit a large percent of students - only 10 to 20 percent are likely to find a better school option (see the full report here).

Above, Dianne Piché, Executive Director of the Citizens' Commission on Civil Rights and longtime civil rights advocate, responds to our report and offers her insight on what it means for the discussion on desegregation and school choice. Thanks to Dianne for her thoughtful comments--I expect that they will spark some good discussion about our conclusions and what those conclusions might mean for the future of interdistrict school choice.

Sock Puppet

I had just assumed that when well-respected publications like Education Week decide to allow people to write anonymously under their good name, they make it clear that you can't do stuff like this. Apparently not.

Update: As I should have noted above, the post in question looks like it went up before the blog migrated to Ed Week. So they may very well have a "no sock puppet" rule.

The Schools We Need, Continued

A few additional thoughts on the post below about margins of error, etc., based on the comments in this and earlier posts. (An aside: My take on anonymous commenting is pretty much akin to my take on anonymous blogging: Why? It's a free country, nobody's going to yank you out of your house in the middle of the night because of what you write about No Child Left Behind. Identify yourself, we'll all be better for it. Particularly, I should add, if you're claiming some kind of expertise. If you're really a statistics professor, say so; I'm not giving handles like "StatsProf" any more credence than "Anonymous" or "SomeGuyYellingOnTheStreetCorner.")

First, some people have raised the issue of measurement error vis a vis the tests themselves. And it's true that (A) tests only cover a fraction of what we want to students to know, and (B) tests are an imperfect measure of that fraction. But the place to make allowances for that should be the process of setting cut scores. States don't require students to get 100 percent of the answers right to be deemed "proficient." (In fact, all states have a separate, higher "advanced" level which is also less than 100 percent.) Once states determine what less-than-100 score is proficient, it makes sense to build in some cushion to account for imperfections in the instrument, random variation, etc. This is particularly true if the test is being used for individual high-stakes purposes, i.e. graduation and grade retention policies. But having made those allowances, don't layer on new allowances with confidence intervals and tell the public that students are meeting NCLB performance thresholds when in fact they're not.

Second, there's certainly no unanimity in academia on the issue of confidence intervals. Some very qualified, well-trained people see it one way, some see it another.

Third, what's consistently lost in all of these discussions is the need to balance the risks of Type I and Type II errors. When implementing NCLB over the last six years, states have been almost exclusively interested in making sure that good schools aren't mistakenly identified as "in need of improvement" or "failing" under NCLB. And that's a legitimate thing to worry about. But they've all but ignored the risk of under-performing schools being mistakenly not identified as needing improvement. There's no way to completely eliminate both risks; states have to find a reasonable middle ground between them. Instead, many states have been obsessed with finding new ways to make sure there is absolutely no possible way for a good school to be falsely identified as underperforming, and in doing so have knowingly let schools and districts that are obviously not serving students well slide by for years and years. Of course, focusing only on minimizing the first kind of risk makes sense if you view NCLB as fundamentally illegitimate and malign, but if that's the case just say so and stop pretending your concerns are limited to proper use of statistics.

The Schools We Have, or the Schools We Need

Last week I wrote about the practice of applying statistical margins of error to the percent of students in a school who pass a test. It's a goofy idea, I said, because: "unlike opinion polls, NCLB doesn't test a sample of students. It tests all students. The only way states can even justify using MOEs in the first place is with the strange assertion that the entire population of a school is a sample, of some larger universe of imaginary children who could have taken the test, theoretically. In other words, the message to parents is "Yes, it is true that your children didn't learn very much this year, but we're pretty sure, statistically speaking, that had we instead been teaching another group of children who do not actually exist, they'd have done fine. So there's nothing to worry about."

Eduwonkette responded with her own take on the issue by cutting and pasting from Harvard professor Dan Koretz's recent book on educational assessment, in which he says:

This question was a matter of debate among members of the profession only a few years ago, but it is now generally agreed that sampling error is indeed a problem even if every student is tested. The reason is the nature of the inference based on scores. If the inference pertaining to each school...were about the particular students in that school at that time, sampling error would not be an issue, because almost all of them were tested. That is, sampling would not be a concern if people were using scores to reach conclusions such as "the fourth-graders who happened to be in this school in 2000 scored higher than the particular group of students who happened to be enrolled in 1999." In practice, however, users of scores rarely care about this. Rather, they are interested in conclusions about the performance of schools. For the inferences, each successive cohort of students enrolling in the school is just another small sample of the students who might possibly enroll, just as the people interviewed for one poll are a small sample of those who might have been.

In one sense Koretz frames the issue correctly; this is indeed all about inference. But he's wrong to imply that this is a methodological question best resolved by the consensus opinion of statisticians and assessment professionals. It's not. Rather, this really goes to the heart of how we conceive of public schools, what we expect of them, and how we hold them accountable for student learning.

By one of way of thinking, school outcomes are the result of a production function in which schools are constant and inputs, including students, vary. In other words, the school is what it is, with a certain amount of resources, certain number of teachers with certain qualifications, along with curricula, facilities, etc. Every year, the inputs (students) change. If, in one year, a brilliant group of students comes through the door, outputs (test scores) go up. If a very tough group arrives in the following year, outputs go down. If this is how you see things, then Koretz's analysis makes sense and sampling-based margins of error are fair--indeed, absolutely necessary.

The problem with this idea is that it assumes that schools are inflexible and un-improvable, inert black boxes that serve as little more than conduits for inputs and outputs. Accountability policies assume something else: that both students and schools can vary. That schools can, and must, change they way they teach to fit the particular needs of their students in a given year. If a cohort of particularly difficult students enters the system, and this becomes apparent as they move through first and second grade, by the time they hit the tested grades the school or district needs to reallocate resources and plan accordingly.

Statisticians have a weakness for black box thinking, because the box contains a lot of things that are essentially unmeasurable. You can't put a number on the relationship between a principal and her teachers, the quality of teamwork, the level of commitment and hard work among the staff, the sophistication and flexibility of the instructional plan. You can, however, put a number on student learning--an imperfect one, to be sure, but close enough to render reasonable judgments about school success.

Schools need to be organized, staffed, led, funded--and held accountable--for the performance of the students they have, not those they might have had, or wish they had, or had once or may have again. These students, this year, are the ones who matter. That conviction, and those that follow, are much more than a matter of statistics.