USA vs. The World

United States schoolchildren are not the highest performing in the world, on average. This is well known and constantly cited in various calls-to-arms, from the memorable "hostile foreign power" rhetoric of A Nation at Risk to garden-variety speeches warning of economic threats from brainy children in Beijing and Bangalore. The track record is spotty, to be honest--remember when the 240-day Japanese school year was going to lead to total American subservience under the yoke of the Rising Sun by the mid-1990s? There's also plenty of controversy over tests and methods. But the underlying point seems fairly indisputable--children in some other countries learn more. For example, here's how things look on the 2007 TIMSS 4th grade math test:



We do okay, indistinguishable from the mean among OECD countries and better than the average of all countries, but substantially worse than Hong Kong, Singapore, Taiwan, Japan, Russia, England and a few others. Eighth grade scores look much the same.

But the tricky thing about looking at average performance in the United States is that our education system is unusually large, diverse, and decentralized. Parts of it are really good. Other parts are shamefully bad. And in a number of important respects, we can only improve the system part by part. So it's worth knowing just how well those parts are doing. Thankfully, Gary Philipps of the American Institutes of research has done a service by converting state and city-level scores on the NAEP to TIMSS equivalents. Here's what he found:



Turns out that a few of our states are on par with the world's highest performing countries when it comes to educational achievement. Massachusetts in particular stands out, and four other states--Minnesota, New Jersey, New Hampshire, and Kansas--received grades of "B," up there with the likes of Japan. On the flip side, there were a bunch of C's and one D+ in, of course, Washington, DC, where fourth graders learn math at the same level as Ukraine.

This is useful information. International comparisons are often shot down on grounds of fundamental non-comparability. After all, Singapore and Hong Kong are tiny little bits of Asia that just happened to have been sequestered into autonomous political entities by the British because they were advantageously located for international commerce. Countries like Japan and Finland (which tops the PISA test but doesn't participate in TIMSS) have unusually homogeneous populations and strong cultural ties among citizens as well as other beneficial non-education factors--strong social safety nets, low crime, school-oriented cultures, etc. They're just not like us, the thinking goes, so it's unreasonable to compare us to them.

But New Jersey isn't an autocratic city-state on the tip of the Malay peninsula or a Nordic socialist paradise or anything like that. Nor is Massachusetts (well, maybe the socialist part) or Minnesota or New Hampshire or Kansas. They're all medium-sized states in America, subject to American laws, filled with lots of Americans in all the diversity that makes this nation great. Massachusetts in particular, the highest performing state, is full of people from all manner of racial, ethnic, religious, and economic backgrounds. It has relatively high business taxes and relatively good social services compared to other American states but it's far from France or Finland or Japan.

One the hand, this should make us optimistic. American schools systems can in fact compete with the world's best--some of them measure up very well right now. One the other hand, we should be sobered and far less willing to explain away the inadequacies of our worst-performing states on the grounds of vast, irreconcilable differences of politics and culture.

How To Not Waste $2.5 Billion

President Obama has proposed creating a $2.5 billion "College Access and Completion Fund." This is a terrific idea, given the huge problems we have (see here and here) with college completion. But there are good and bad ways to spend $2.5 billion over five years. As Congress considers the proposal, it should keep them in mind:

Bad Ways

1) Paying off state guarantee agencies. The $2.5 billion will come from savings realized by cutting the middlemen out of the federal student loan program. Many of those middlemen are, unsurprisingly, opposed to this idea. While lending giants like Sallie Mae and huge, world-destroying banks have gotten most of the attention, a bunch of ostensibly non-profit agencies also stand to lose out, including state guarantee agencies, which occupy an arcane and largely vestigial role in the loan process. If they're still going to get paid, it should be for something of actual value, like servicing government loans. Giving them a chunk of this money for ill-defined "counseling" purposes or whatnot would waste scarce resources. 

2) Straight formula distributions. The easiest and most politically expedient way to divvy up this money is via a simple funding formula: every college gets an amount equal to their share of all Pell grant students or something similar. Formulas are objective, consistent, easy to explain, and guarantee that nearly every Congressional district gets a taste. They're also a surefire method of ensuring that dollars do little good. The federal government has a long, ignoble history of distributing K-12 funding this way, spreading Title I allocations that amount to only a small fraction of total education spending far and wide to nearly every school district in the nation with little attention to need or whether the money actually does any good. As a result, much of it did little good. There have been improvements to the Title I formulas in recent years, but much of the money is still shot out via a method that (no lie) gives anti-poverty money to schools in Beverly Hills. Formulas are like bamboo: once they take root they're nearly impossible to eradicate. Let's not make the same mistake twice.

3) Data systems, college preparation, and other things that seem perfectly reasonable but aren't actually about directly helping college students earn degrees. Some people have proposed giving states and institutions wide latitude in spending this money, including building data systems to track completion. But there's money elsewhere in the federal budget for that, as well as for improving college preparation through programs like GEAR-UP, etc., etc. This money should be for directly helping college students complete college.

Good Ways

1) Competitive Grants.  Some states and institutions are well-positioned to use this money. They have solid programs in place, good people on the ground, and accountability systems that track success.  Other states and institutions have no idea, but will be happy to cash the check, hire a new administrator, and shuffle the rest of the money around behind the scenes to use for the things they actually care about. Per above, formulas by definition make no distinction between the institutions that are best prepared to use funding and those that are least prepared. $500 million a year isn't much compared to the $400 billion we spend on higher education annually. If this money isn't focused on those who can spend it well, it will be wasted, and students will bear the brunt of that failure. 

2) Partnerships: There's a tricky balance to strike in any grant program. On the one hand, it's madness to try and legislate the who's and how's of a specific completion initiative. Colleges have diverse missions and student bodies--the best approach at a small community college might be entirely different than the most successful strategy at a big research university. But dispersing the money to thousands of disconnected efforts, each trying to independently re-invent the wheel, isn't a good idea either. Preference should be given to coalitions of institutions, systems, or even states that are prepared to help and support one another in pursuing larger completion goals over multiple years. 

3) Accountability. The good thing about college completion is that it's relatively easy to measure. Funds should be distributed with the understanding that grantees will need to show improved results if they expect to come back for more. Completion numbers should be broken down by students' race/ethnicity, gender, and income status. Part-time and non-traditional students should be included. Students should be followed along extended time horizons as they move and transfer. If someone enrolls in a community college part-time for a couple of years, earns 30 credits, transfers to a four-year institution and ultimately earns a degree, that's a success for all concerned. Grantees should be evaluated for effectiveness and efficiency. If you walked up to a man on the street and said, "Hey, if you complete college within six years, I'll give you $500 million," that would probably work. But it wouldn't be a good use of taxpayers' dollars. 

4) Evaluation. Typically, government programs are evaluated as follows: A) Enact program. B) Worry about other things for awhile. C) Come back a few years later to consider re-authorizing program. D) Try to figure out if program worked. But by then it's far too late--to properly evaluate a program, evaluators needs to be involved up-front. Researchers should be hired from Day One to study all the different ways the funding was used and figure what worked best. 

Iowa's Charter Schools

I'm proud to say I attended Iowa public schools from kindergarten through college, and it so happens that my education almost perfectly corresponds to the heyday of Iowa's education system.

In 1992, when I was 8, Iowa's fourth-graders scored higher than all but one state in math and all but four states in reading on the National Assessment of Educational Progress. Unfortunately, today's Iowa's children face a gloomier future than I did. Over a 15-year period ending in 2007, only three states had lower academic gains than Iowa, and Iowa now trails 14 states in both subjects.

These data are important to consider against the backdrop of a recent Des Moines Register piece on charter schools in the state. It found little drive for innovation in Iowa's public-school system, no surprise given its ranking from the Center for Education Reform that placed its charter law 40th out of 41 states.

Iowa's charter law earned this low ranking by placing an artificial cap on the number of charters that could open in the state and limited charter-school management to current districts only. Rather than accommodating growth and demand, Iowa's 370 school districts were arbitrarily limited to a total of 20 charter schools. Moreover, the law limits each district to only one charter school, so Des Moines, which enrolls 29,000 students, is limited to the same number of charters as Dows, which enrolls 65.

The second problem is the law's provision that only current districts can open charters. This is the primary reason Iowa's charters have struggled to innovate. They're run by traditional school districts, so it's no surprise they look more or less like traditional public schools. The most successful and innovative nonprofit charter networks operating in other states - such as the Knowledge Is Power Program (KIPP), Achievement First or Green Dot - are legally barred from opening schools in Iowa. This is a major loss for Iowa's children.

More charter schools will not be a panacea for Iowa's schools, and the evidence on their effectiveness remains mixed (even after yesterday). To read more about Iowa's law and my suggestions for improvement, read my op-ed in today's Register.

Do Full-Time Faculty Help Students Complete College?

This morning I moderated a panel discussion at the Library of Congress focused on college completion. When we came to the Q&A, Cary Nelson, President of the American Association of University Professors, posed a question (I'm paraphrasing from memory):

"One thing nobody on the panel has mentioned is that fact that colleges with higher completion rates also have a larger percentage of their classes taught by full-time professors. So that's one thing we could do: give colleges the resources to employ a stable, full-time faculty."

There are some obvious correlation / causation issues to resolve here. Because full-time faculty are more expensive than contingent faculty, the colleges that tend to employ a lot of them tend to be wealthier than those that don't. Wealthy colleges also tend to enroll a disproportionate number of wealthy, academically well-prepared students, who are more like to complete college. So yes, colleges with stellar college graduation rates are more likely to hire full-time, well-credentialed, tenure-tack professors to teach. But they're also more likely to have lots and lots of other things that also independently improve graduation rates. Resource advantages in higher education tend to be highly co-linear.

So I'm curious: Is there any good research out there that properly explores the relationship between full-time / tenure track status and student outcomes like retention, completion, and learning? Most of what I've seen on the subject only speaks to things like student - faculty interaction and doesn't really get to outcomes. The answer seems non-obvious to me: one the one hand there seem to be obvious advantages to being taught be experienced, knowledgeable professionals who are well-integrated into the university community; on the other hand tenure-track faculty are subject to some pretty severe professional incentives related to publishing that actively push against the time available for helping students learn, earn degrees, etc.

More broadly, given that our vast, world-beating higher education system is populated with many thousands of people who have been highly trained to unravel complex phenomena, and that the subjects in questions aren't located in some distant land nor are they indiscernible without complex scientific equipment but rather are right there on the campuses where all of our researchers live and work, and this is a subject that clearly arouses a lot of strong feelings and is thus in sore need of more empiricism, why isn't there more research in this area?

The New Charter School Study

Macke Raymond, the lead author* for CREDO's new study of charter school performance, emphasized on yesterday's conference call about the report the importance of the word "variability" when discussing the study's results. And there is a lot--among schools, among states, and among the students. In the end, the study (as most good research does) raises more questions than it answers.

The top-line result from the study, and the one most likely to get press attention, is that charter schools are not performing as well as equivalent traditional public schools--17 percent of the charter schools outperformed their traditional public school equivalents, 46 percent were indistinquishable, and a disturbing 37 percent performed significantly worse. That result isn't great for charter advocates - 15 years into charter schooling and one would hope that aggregate analysis of charter school performance would at least be on-par with traditional public schools, if not slightly better.

But that one result doesn't really tell the story of charter school performance--instead, it is that key word "variability" that starts to get at what is happening.

Charter elementary and middle schools actually performed better than their traditional public school peers overall, while high schools and multi-grade schools did worse. Black and Hispanic students showed significantly lower gains than their matched traditional public school students. But low-income and English Language Learner students posted larger gains than their traditional public school peers.

And then there is the variation among states. The report examined results from 16 states and found that in math, for example, 5 states showed higher gains among charter school students: Illinois (Chicago), Colorado (Denver), Louisiana, Missouri, and Arkansas. In 9 other states, charter school students performed worse, including Arizona, California, Florida, Georgia, Minnesota, New Mexico, North Carolina, Ohio and Texas.

The CREDO researchers also drew some conclusions about state charter school policy from this variation in state results. They looked at state caps on the number of charter schools, the availability of multiple school authorizers, and whether the state has an appeals process for new charter proposals. Both the presence of caps and availability of multiple authorizers were associated with lower charter school performance, while an appeals process was associated with higher performance. The results for multiple authorizers is surprising and runs counter to the thinking of many organizations, including Education Sector. Perhaps 'multiple authorizers' isn't the best way to frame a state's policy--instead the difference is likely in how well authorizers are held accountable for their work and whether the state has statewide, professional authorizers that are able to focus sufficient resources and attention to the job of monitoring school quality.

In the end, this report is a good discussion starter. Why would charter schools in Louisiana show significantly positive growth while charter schools in Texas show significantly negative growth? Why would charter schools serving elementary and middle grades separately do better than traditional public schools, while charter schools serving those grades under one roof do worse? And what is it about charter schools that is beneficial to low-income and English Language Learner students, but isn't for Black and Hispanic students?

I'm looking forward to the next report, which promises to dive into some of these questions. In the meantime, the policy recommendations from this first report underscores that the current shift in discussions of charter school policy, from a focus on the quantity of charter schools to emphasizing the quality of those schools, is precisely where the charter school movement needs to go.

* And ES board Vice-Chair

Arts, Continued

Per Chad below, new results from the NAEP 8th grade Arts & Music test show very little change over time. So little change that's it's really kind of fascinating. In 1997, respondents got 42 percent of the art questions right. In 2008, they also got 42 percent of the questions right. Of the 12 listed questions / tasks, there was no statistically significant change on 11 of them. The only difference: "Identify a compositional feature of a medieval artwork" dropped from 39 percent to 32 percent. Music was a little worse but still very similar. Overall the, the percent correct dropped from 53 to 51 percent, a small but statistically significant difference. Of the 21 questions / tasks, there were statistically significant changes in only six -- five down, one up. All of this is with the caveat that only the multiple choice questions are comparable across years--the constructed response portions of the tests are not.

A lot has changed since 1997. Eighth graders then were only dipping their toes into the Internet then; now they don't know anything else. And of course, eighth graders in 2008 bore the full brunt / enjoyed the full benefit of No Child Left Behind, having completed six consecutive years in which their schools were tested and rated under NCLB. During that time, roughly 6.8 billion articles and news stories were published stating unequivocally that NCLB is responsible for a drastic narrowing of the curriculum, arts teachers being taunted by students and beaten up in the faculty lounge, etc., etc. And yet arts and music scores stayed virtually the same.

I'm not even sure these unchanged numbers are obviously a good thing. Shouldn't there have been some tradeoffs, some substitution of results in focused-upon subjects like reading and math for everything else? Math results for 13-year olds got better during roughly the same time period while reading results stayed flat. In some ways this is all an argument for humility among federal policymakers in their aspirations for magnitude and pace in changing a massive, decentralized K-12 education system.

New Report: NCLB Did Not Narrow Arts Curriculum

Under the headline, "Frequency of arts instruction remains steady, " a new report from the National Assessment of Education Progress concluded:

In 2008, fifty-seven percent of eighth-graders attended schools where music instruction was offered at least three or four times a week, and 47 percent attended schools where visual arts instruction was offered at least as often. There were no statistically significant changes since 1997 in the percentages of students attending schools offering instruction in music or visual arts with varying frequency.

There were also no significant differences found between the percentages of students in different racial/ethnic or gender groups attending schools with varying opportunities for instruction in either music or visual arts in 2008.

Does this mean the NCLB-negatively-impacted-arts-curriculum meme is done? Perception feeds reality, except when facts rear their ugly head.