Best of 2007 (Colleague Edition)

Continuing in the "Best of" spirit of the times, here's the list of 2007's best songs from Daria Hall, Assistant Director for K-12 Policy at the Education Trust. Daria knows substantially more about NCLB implementation than I do, a fact I note in sympathy as much as in admiration. Daria is also trusted source of new music recommendations, so--as with NCLB policy--similarity between this and previous posts is generally a function of my availing myself of her expertise and not the other way around.

Keep the Car Running- Arcade Fire
Fans- Kings of Leon
Imitosis- Andrew Bird
Impossible Germany- Wilco
Paper Planes- M.I.A.
All the Old Showstoppers- New Pornographers
Breakin’ Up- Rilo Kiley
Gone, Gone, Gone (Done Moved On)- Robert Plant and Allison Krauss
Nude- Radiohead
I Feel It All- Feist
Can You Please Crawl Out Your Window- The Hold Steady (I’m Not There Soundtrack)
Stronger- Kanye West
Pagan Angel and a Borrowed Car- Iron & Wine
You Got Yr. Cherry Bomb- Spoon
Rehab- Amy Winehouse

The Testing Quandary

ES Co-Director Tom Toch answers a tough test question posed by the NYT's Freakonomics blog:

Should there be less standardized testing in the current school system, or more? Should all schools, including colleges, institute exit exams?

Other respondents include Fair Test's Monty Neill and Gaston Caperton, the President of the College Board (for some reason, he really likes the SAT...). Read all of the responses here.

It's (Not) So Easy

The Post ran a useful article a few days ago ("Calls Grow for a Broader Yardstick For Schools") about one of the more interesting challenges of NCLB reauthorization and education policy generally: expanding the scope of educational accountability beyond standardized test scores in reading and math to include many other important things--mastery of other subjects, more generalized abilities like critical thinking and analytic reasoning, "soft" skills like leadership and teamwork, graduation and success in college and the workforce, etc. etc. But it leaves the central question answered: if nearly everyone thinks this is a good idea, why aren't we doing it already ?

Mostly, I think, because this imperative bumps up against other imperatives, and nobody has figured how to adequately reconcile their inherent conflicts.

NCLB was designd to make the assessment of schools objective, universal, and unavoidable. In other words, all schools within a state are assessed according to the same standards and in the same way. There are very good reasons to put a premium on this. Without objectivity and universality, it's a short road back to the default judgment that most schools render upon themselves: "We're doing the best we can, given the students we have." Given how badly some schools and students are failing, that's just not good enough.

The way that NCLB achieves objectivity and universality, however, has significant shortcomings. It's a mechanistic process, based on rules intead of human judgment. The problem is that it's really hard to developing rules that (A) accurately assess something as complicated as a school and (B) people can understand.

For example, here's a by-no-means-exhaustive list of some of the important categories of information we might want to gather about a school and its students, along with the number of possible values for each:

Student Race/Ethnicity: 5 (White, Blank, Hispanic, Asian, Indian)
Student Gender: 2 (Male, Female)
Student LEP status: 2 (Yes, No)
Student Disability status: 2 (Yes, No)
Student Economic status: 2 (Low-income, Not Low-Income)
Student Gifted status: 2 (Yes, No)
Subjects: 5 (Math, Reading/LA, Social Studies, Science, Art/Music)
Proficiency Status: 4 (Below Basic, Basic, Proficient, Advanced)
Value-Added Growth: 3 (Below Expected, Expected, Above Expected)
Attainment (i.e. graduation, progression to next year in school): 2 (Yes, No)
Grades: 4 (Typical grade configuration)
Timeframe: 4 (Now, Short-Term, Medium-Term, Long-Term)

Keep in mind that this is, in many ways, a very conservative estimate of the number of possible variables. There are more than five significant racial/ethnic groups, more than five important academic subjects, more than four potential levels of academic proficiency or value-added growth to consider, far more than one category of disability, gifted status, etc. Heck, you could even argue about gender.

But even this highly simplified model produces 307,200 possible outcomes. Each of them tells us something different, and as such could theoretically merit a different response. This throws the decisions of NCLB's authors into a fairly sympathetic light. They knew that a 307,200-element accountability system wouldn’t fly, so they started narrowing things down.: Two subjects and five racial/ethnic categories. One category each for LEP, special ed, and economic status, but no combinations—in other words, we measure the performance of white children and low-income children, but not white low-income children. Gender and gifted status are out. One proficiency level, no value-added. Only seven of 12 grades, and multiple grades can be combined. Include one growth measure (safe harbor), but make it either/or so you don’t create extra variables. Each of the 16 distinct outcomes (two subjects X eight student categories, although very few schools will have all eight) has equal and overriding status as an indicator of school success. Miss one, miss all, it doesn’t matter—your identification as not making AYP is the same. Then, having rolled the entirety of a school’s success into the single binary AYP variable, put it on a four-level time scale: 0-1 years in a row, 2-3, 4-5, 6 or more. Each level corresponds to a collection of mandated and optional responses—None, In Need of Improvement, Corrective Action, Alternative Governance.

Even this relatively small level of complexity seems barely manageable. The testing industry is popping rivets trying to handle two academic subjects in seven grades. State Department of Educations struggle, some mightily, to gather all the required data and turn school ratings around on time. There are constant complaints about the expense and bureaucracy of compliance and time lost to preparing for and taking one test in two subjects per year.

And even with all the compromises and simplifications, most people still don't know how the NCLB system actually works. That diminishes the capacity of the law to act as a catalyst for change, since educators and policymakers can't constructively respond to signals they don't understand.

The article quotes Ed Trust's Amy Wilkins saying maybe this is okay: "Proponents of multiple measures say it will give a richer, fuller view of a school, but this isn't about a rich view of a school. It's about failures in fundamental gate-keeping subject areas." That view reflects Title I's origins in and continued focus on compensatory education for low-income children, and I agree this needs to remain the first priority. But like it or not, NCLB has come to be about all students and all schools, and that demands a richer view on some inescapable level.

Adding more information to the existing rules-based system will consume even more scarce resources and create even more hard-to-manage complexity. Not adding more information will leave us with an accountability system that reflects only a fraction of what we want for schools. That argues for a non-rules-based approach, one that relies more heavily on human judgment, since people are much better at making sense of vast amounts of information from disparate sources than rules. But that, in turn, threatens universality and objectivity. Perhaps one could mitigate this problem by aggregating many judgments through more robust market-focused systems, but then we're opening up a whole new can of worms...

Anyway, it's tricky. Anyone who thinks the shortcomings of the existing system are a result of obvious choices not made should think again.

Best of 2007 (Wife Edition)

In addition to being smart, beautiful, and possessing the most spouse-friendly hobby imaginable (cooking), my wife Maureen also has fine taste in music. She looks at blogging with suspicion, particularly having read this and last year's special music festival posts and noting that at least of half of the insightful critical opinions and observations, such as they are, were hers, not mine. I tried citing the timed-honored principle of spousal communal ownership of intellectual property, but she didn't buy it. So, here's Maureen's iPod "Best of 2007" playlist:

Arcade Fire, "(Antichrist Television Blues)"
The National, "Apartment Story"
Radiohead, "Bodysnatchers"
LCD Soundsystem, "North American Scum"
Band of Horses, "Is There A Ghost"
Wilco, "Impossible Germany"
Bloc Party, "Hunting for Witches"
Spoon, "You Got Yr. Cherry Bomb"
The White Stripes, "Icky Thump"
Ted Leo & the Pharmacists, "The Sons of Cain"
Rilo Kiley, "Silver Lining"
Iron & Wine, "Boy With a Coin"
The New Pornographers, "Myriad Harbour"
Stars, "Take Me to the Riot"
The Apples in Stereo, "Can You Feel It?"
White Rabbits, "While We Go Dancing"
Andrew Bird, "Fiery Crash"
Buffalo Tom, "You'll Never Catch Him"

I might have gone with different selections in some cases-- "Woman King " or "Jezebel" for the Iron & Wine song, for example -- but overall this is a fine way to spend 90 minutes driving to your parent's house for the holidays.

"Best of" lists are often a good way to identify your critical / genre weak spots; I bought The Hold Steady's "Boys and Girls in America" a year ago solely because it was on top of a lot of Ten Best lists from respectable sources. On first listen I was like "Not bad," on second, "Pretty good," on fifth, "Kind of great!" and so on and so forth. By the time I saw them at the 9:30 club last month I was three/fourths convinced they're authentic American geniuses; it was certainly one of the ten best concerts I've ever seen.

Fire This Time

Last week the Post reported the following:

A technology manager for District schools who stuck schoolchildren with his tabs for thousands of dollars worth of lavish restaurant meals, nightclub jaunts and a visit to a strip club was charged yesterday with filing fraudulent expense reimbursement requests.

The story ran in the Metro section, below the fold. That's because municipal corruption in DC is like lake effect snow in Cleveland--the fact of it isn't news, only the magnitude. Compared to the property tax office, teachers union, and charter school board officials who stole hundreds of thousands or millions of dollars, this guy is the equivalent of a small flurry.

I note this in light of the ongoing efforts of DC mayor Adrien Fenty and schools chancellor Michelle Rhee to get legislation through the City Council that would allow Rhee to fire central office employees "without cause," which in gov-speak means "if she thinks they're doing a bad job." Fenty and Rhee are getting a fair amount of pushback from the council and the civil service and teachers unions, but it looks like the plan will go through.

To be clear, even the current sclerotic civil service system can manage to fire convicted felons. Similarly, "Work hard or I'm going to fire your a--" is not, in and of itself, a comprehensive management strategy. It's equally important to identify and support high-performing--or even normal-performing--employees, to create a constructive, collaborative work environment that attracts talent.

That said, incompetence and corruption are two sides of the same coin, flowing from a generalized organizational culture of indifference--or even opposition--to things like excellence, accountability, and service to the public. That's what Fenty and Rhee are after, I think, and given the catastrophic failure of the system they're trying to change, they deserve a great deal of latitude in how they try to get there.

"School Choice on Steroids"

NPR’s Larry Abramson takes a trip to Mapleton school district outside of Denver, Colorado, where they are “trying to offer school choice on steroids.” But (unlike in baseball) this isn’t magic. Test scores are still low, and the idea of school choice hasn’t fully permeated the mindset of students and parents in Mapleton.

Despite offering everything from a hands-on, project-based school modeled after New Tech High in Napa, California to an International Baccalaureate school complete with uniforms, many parents still base their choice decisions on convenience. Many of the school improvements promised by choice theorists depend on parents choosing schools with the best academics. But reality shows that parents and students make decisions based on a host of other factors—where their friends go, how close the school is to home or work, and some very important, if not academic, criteria, like school safety.

The research and advocacy around school choice often focuses on building the supply of schools and reducing barriers to choice for parents and students, but not much has focused on what is needed to build a knowledgeable consumer base. There is a bit of a “build it and they will come” attitude about school choice reform. But without a culture of informed choice, school choice reforms might either see little to no impact, or might see choice (somewhat like the higher education market) shifting priorities to things like sports or fancy facilities and away from the quality of teaching and learning in schools.

Listen to the segment here. And if you want more NPR segments on school choice, check out these two pieces—one on Green Dot Public Schools’ takeover of Locke High School in Los Angeles, and the other profiling Green Dot’s founder, Steve Barr.

The Post Fails Math

One of the ways you know a particular story has really embedded itself in the public consciousness is when it becomes a genre. As in the "People who think public education should teach X blame NCLB for reduced attention, resources, etc. for X" story, of which roughly 632,000 have been published in major media outlets in the last five years. Today's version in the Post focuses on music education, and contains a significant error. It says:

Despite research showing that students who study music have better attendance, achievement and lifetime earnings, music classes are struggling to survive. Supporters of such classes place some of the blame on the federal No Child Left Behind law....As instructional time in math, language arts and other subjects students must achieve proficiency in has risen, time devoted to other subjects has declined. Time spent on arts and music in 2007 is about half what it was before No Child Left Behind became law in 2002, according to a report recently released by the nonprofit Center on Education Policy.

The sentence I've italicized above is not even close to being right.

You can read the Center on Education Policy report referred to in the article here. It only shows detailed data for changes in instructional time for elementary schools. According to the report, the districts surveyed devoted 110 minutes per week to art and music in 2007. Districts that reduced time for art music reported an average reduction of 57 minutes. 110 minutes is not "about half" of 167 minutes, it's 65.8%, a hair below two-thirds.

Much more importantly, only 16 percent of districts reported reducing time for art and music at all.

Needless to say, this is a hugely important distinction. The plain meaning of the sentence printed in the Post is that NCLB has caused a 50% reduction in the total time spent teaching art and music. The report suggests that the actual number is closer to five percent (.34 X .16). In other words, the story is wrong by as much as an order of magnitude.

This seems like a classic symptom of the objectivity / accuracy value imbalance in the media -- newspapers will go to great lengths to make sure that a political reporter's spouse's friend's cousin's college roomate didn't buy lemonade from a stand in front of the candidate's neighbor's house, yet they make math mistakes like this all the time.

A More Selective Pool of Teachers

ETS released a new research report on teacher quality last week showing that the teacher pool seems to be improving, at least on academic measures. The researchers compared the 1994-1997 and 2002-2005 Praxis test takers and found that the SAT scores and GPAs of teacher candidates have increased from one cohort to the next. Why? They credit a combination of policies, both federal and state, including alternative pathways to teaching, higher standards (and standards in general) for academic qualifications, and tougher admissions and licensing requirements. The research suggests that it's a more selective process now–minimum passing scores have been raised and, as a result, Praxis passing rates are down. And the proportion of test takers with prior teaching experience has risen, suggesting a change in the traditional undergraduate education degree route to teaching.

Less rosy are the results for elementary, special education and physical education candidates. Except for special education, where there appears to be some decline, these teacher candidates have better grades and scores too. But they still lag far behind their subject-matter-certified peers for secondary education. And the typical teacher, demographically, isn't much different than a decade ago. There are still disproportionately low numbers of African American and Latino teacher candidates (with the K-12 school population now less than 60 percent white, it's a little startling to still see the teacher pool hovering near 90 percent white). And the male-female ratio hasn't budged at all- still holding at only a quarter male).

The research can't say anything about the classroom effectiveness of these teachers. That's a whole other can of worms. But the direction overall- toward a more selective and perhaps respected pool of teachers- seems positive. The next question for researchers, then, is whether there is some state or federal policy that matters most for this change, or if it is some strange combination of policies that make the difference for teacher quality.