Department of Things I Am Sad to Have Been Right About

From a post dated July 12, 2008:

Walking out of an afternoon showing of Wall-E last weekend, I noticed some big cardboard movie displays advertising The Spirit, a forthcoming movie based on the classic Will Eisner comic book series, to be written and directed by Frank Miller. We're clearly living in a Frank Miller heyday, and it's been a long time coming...[four-paragraph summary of Miller's career]...but to be honest I'm worried that Miller's descent into over-stylization and self-parody, both visually and verbally, is too deep to reverse, and that he'll end up crashing and burning at the very moment when decades of influential work are finally bringing him fame and fortune."

From The Onion AV Club, this week:

In comics, it took Miller decades to devolve into embarrassing self-parody. In film, he’s made that leap over the course of a single disastrous film. A.V. Club Rating: D

From the New York Times:

To ask why anything happens in Frank Miller’s sludgy, hyper-stylized adaptation of a fabled comic book series by Will Eisner may be an exercise in futility. The only halfway interesting question is why the thing exists at all...a talky, pretentious stew of film noir poses and crime-fighter clichés.

Etc., etc. It's a shame.

Crisis Averted

In May the Center on Education Policy (CEP) released a report looking at how states structured their Annual Measurable Objectives (AMOs). The No Child Left Behind Act required only that AMOs reach 100% by 2014 and that each increase must be equivalent, and it allowed states up to three years of no growth. It being 2002 at the time, about half the states chose to backload their AMOs, calling for no gains in the early years of the law, followed by steep increases each year leading into 2014. Depending on your cynicism, this was either to allow districts the opportunity to prepare for the new requirements or a way to force changes to the next generation of the federal education law.

The May CEP report warned of the impending consequences of such backloading:

Although states may have had logical reasons for choosing a backloaded approach, it appears that schools and districts in backloading states are likely to have more difficulty making AYP than in previous years, and the number of schools identified for NCLB improvement in these states might rise.

The numbers are now in ($) for 46 states and the District of Columbia, and it turns out the CEP report was wrong. Nationwide, 7.3 percent more schools failed to make Adequate Yearly Progress (AYP) this year, for a total of 35.6 percent of all Title I schools. But, that increase was led by the states categorized by CEP as "incremental." While backloaded states averaged only a 3.2 percent increase, incremental states rose 7.7 percent. The percent of schools rated "in need of improvement" rose 2.1 percent for a total of 17.9. This too was led by states that were incremental in their AMOs.

These results suggest that Charlie Barone was right: the safe harbor provision is working. Safe harbor allows districts and schools to make AYP so long as they reduce by 10 percent the percentage of students in any sub-group not meeting proficiency targets. Backloading states are clearly benefiting from this provision (which Charlie dubbed the "poor man's growth model"), and this is entirely a good thing. The law's built-in flexibility is being put to work. The fear that 100 percent of schools will fail is not happening, and the sky remains intact.

The Gladwell/ Kane Theory of Teacher Recruitment

Overheard: a business CEO will hire any Harvard MBA before they even begin the program. It isn't the education itself that makes them valuable employees, in this estimation, it's the screen that let them in that proves their quality. In education, it turns out all of our traditional screens, and even some untraditional ones, don't tell us much about how effective the incoming teacher will be.

In a recent report for the National Bureau of Economic Research, Jonah Rockoff, Brian Jacob, Thomas Kane, and Douglas Staiger look at a host of teacher recruit characteristics and analyze their ability to predict the teacher's effectiveness in the classroom based on these characteristics. The characteristics include some commonly studied ones like their SAT scores, whether the teacher passed their licensure test on the first try, their undergraduate major, and the selectivity of their undergraduate college. The list also included less commonly used measures like tests of cognitive and mathematic ability, conscientousness, extraversion, and efficacy. They also included a commercial screener used by several large urban districts.

The added information did lead to better predictions. But even with all these new variables on incoming teachers, the researchers could explain only 12 percent of the variance in teacher effectiveness. As they wrote, "This underscores the difficult, perhaps impossible, task of identifying systematically the most highly effective or ineffective teachers without any data on actual performance in the classroom."

The findings do not mean this difficult task is impossible (promising research out of Louisiana suggests that teacher preparation programs matter), but it does suggest we take the Malcolm Gladwell / Tom Kane theory of teacher recruitment more seriously. It means that, to get a higher quality teaching workforce, it isn't simply a matter of recruiting more talented, more efficacious, or more extraverted teachers. It means allowing more people to try their hand at the profession, intensively screening them while they're in the classroom, and then enacting salary and personnel policies to both keep them in the profession and keep them performing at a high level.

Unfortunately, we're much more willing to allow screens (really just educated guesses) at the moment of hiring than after the teacher has been in the job. One huge obstacle in basing personnel decisions on teacher effectiveness is being confident that value-added measures reflect actual value and not the effects of one bad class. Some excellent research by Dan Goldhaber and Michael Hansen tested the bad-apple theory in teacher value-added effectiveness. They used three years of data to rank teachers into five groups, and then asked what percent of teachers moved groups. In one dataset, 31 of 281 teachers placed in the bottom group in their students' growth in both math and reading. If we denied these teachers tenure, would we losing great teachers? Or ineffective ones, as the data would predict?

As the chart at left shows, the estimations were by no means perfect. The screen would eliminate some fair and even some very good teachers, but mostly (and this word is not comforting to teachers or their representatives) it screened out ineffective teachers. Mostly the teachers who were ineffective in their first three years were ineffective after.

The word "mostly" should not be reason to summarily dismiss the use of value-added teacher effectiveness scores in personnel decisions. Rather, it should be embraced as a good start, something to be combined with other evaluations, especially given new research suggesting good teaching cannot easily be screened at the hiring stage.

Giving the Game Away

The No Child Left Behind Act is often criticized as creating "perverse incentives" or "unintended consequences" whereby seemingly virtuous policies inadvertently cause more harm than good by incenting bad behavior. It's a convenient man-bites-policy-dog way to frame a news story, and it allows people to preface denunciations of the law with some variant of "Of course I agree with the goals of the NCLB, but..." I've always been puzzled that educators are so quick to argue from the assumed moral weakness of their colleagues, taking as a given that teachers and administrators are bound by various made-up laws of human behavior that compel them to sell little Johnny down the river at the first opportunity. 

That said, the general principle is sound: education is complex and multi-dimensional and accountability systems should reflect that. Incentives should be aligned with goals; if they're not, problems can arise. There's a longstanding concern, for example, that holding high schools accountable for student learning will create incentives for schools to "push out" low-performing students by implicitly or expliciting encouraging them to leave. The solution seems pretty obvious: hold high schools accountable for graduation rates, reducing the temptation to engage in devious push-out behavior. Shut down the easy way out. 

Unfortunately, the high school graduation provisions in NCLB as written are pretty much a joke, allowing many states to adopt insane metrics that bear little or no resemblence to the actual percent of students graduating from high school and/or creating improvement timelines so attentuated that schools wouldn't have to get all students through high school until roughly the launch date of the Starship Enterprise. So the U.S. Department of Education took two eminently reasonable steps by require all states to (1) adopt a common standard of "high school graduation rate" whereby those words actually mean what they say, and (2) create improvement timelines that don't theoretically terminate in the next millenium. 

This would go along way toward solving whatever perverse "push-out" incentives currently exist. It also reflects the explicit policy of nearly every state in the nation, as expressed by their governors in a recent agreement. Naturally, the National Governor's Association supports this policy is working to eviscerate this policy while people are distracted during the upcoming transition. As Charlie Barone reports: 

A reliable source tells us that the NGA is lobbying the Obama transition team to roll back the regulation issued by Secretary Spellings in October that requires states to set a uniform and accurate method for measuring high school graduation rates. Spellings simply put in regs what the Governors themselves pledged to do more than three years ago. However, only 16 states so far have done so.

NGA has targeted a key member of the education transition team to carry their water for them and has been pressing hard, but it is not entirely clear whether the targeted person is helping them.

High school graduation rates represent a useful clarifying issue. There is no doubt that all students need to graduate from high school. There is no doubt that many students don't, and that poor and minority students are less likely to graduate than others. There is no earthly reason why the method for calculating high school graduation rates should vary from state to state, or that it should be anything other than "of those students who begin high school, the percent who graduate." If you're against meaningful accountability for a common high school graduation rate standards, then you're simply against accountability and common standards, period, full stop. 

Investing in the Downturn

Budget cuts and fights to preserve funding will dominate the headlines for at least the next year. But, sometimes, even in a downturn, it's important to invest new funds in particularly promising areas. It's why even in the face of massive financial uncertainty, GM is doing what it can to continue investments in ideas such as the battery-powered Chevy Volt.

A recent article in the Newport News, VA Daily Press gives a good example from education. The article highlights the impressive growth of the Virtual Virginia online learning program. The program offers 22 different AP courses and serves 2,200 students. But, despite the program's success, funding limits capacity and there are wait lists for some courses. And, with looming budget cuts, even the program's current capacity is at risk.

Consider these important facts in the article when thinking about this investment decision:

• The program was designed to serve schools that couldn't afford to hire teachers for AP and other classes.
• More than a quarter of U.S. high school students lack access to advanced courses at their schools, and those at small or rural schools "have the least opportunity to take one or more advanced courses in math, science, English or a foreign language," according to a 2007 NCES report.
• Local districts also save money by not having to hire teachers; Virtual Virginia, for example, only requires districts to pay for textbooks and computer access and assign teachers to monitor students' in-school online sessions.

We don't have enough evidence from the article to run the numbers on the Virtual Virginia program, but it's likely that this type of program is the "Chevy Volt" of public education. It might actually save money. And, it's especially important if we prioritize effectiveness in accomplishing important goals, such as broadening and ensuring equitable access to advanced courses in math, science, and other areas.

The Southern Regional Educational Board, which does study these issues at a much deeper level, just published a thoughtful policy brief making the case for a better, more sustainable funding model for state-run virtual schools. Embedded in the brief is the idea that performance is important--even more so in a downturn.

PS -- Of course, things are so bad at GM that even the Volt is taking a hit.

Journalists and Charter Schools

Eduwonkette has some beef with the Washington Post's recent coverage of charter schools, specifically the Post's claim that public charter schools are outperforming district-run public schools (thanks Chad) on student achievement measures. Accompanying the test score results, the Post reported on the successful practices many schools engage in as reasons for their high scores - many of which wouldn't be possible without the freedom granted to these schools through charter schooling:

With freedom to experiment, the independent, nonprofit charters have emphasized strategies known to help poor children learn -- longer school days, summer and Saturday classes, parent involvement and a cohesive, disciplined culture among staff members and students.

Eduwonkette's complaint is about the accuracy of the Post's comparisons and she reaches back to 2004 and the hubbub that followed the AFT's report, which found that charter schools performed worse than traditional public schools. Eduwonkette's problem seems to be that charter school advocates are happy to take results coming from bad research design (the Post's coverage) so long as they are favorable, but jumped all over the research design of the AFT's report when it came out, even taking out a full page ad in the New York Times.

Sure, advocates are always happy to see results that support their position, but it's not fair or even all that reasonable to compare the Post's journalistic reporting of one city's results with the AFT's research report comparing charter schools and traditional public schools nationwide. The AFT sought to make a judgment on charter school performance across the nation, and made pretty big claims about the results, saying that they, "reinforce years of independent research that show charter schools do no better and often underperform comparable, regular public schools". In contrast, the Post made conclusions about the performance of charter schools in just one city, and, to their credit, included a graphic that shows the variance in charter school performance, rather than just relying on averages to tell the story.

This variance is the most important point in the story - that there is nothing inherent to charter schooling that produces higher student achievement, but, given the flexibility, there are some very concrete things schools can do to dramatically improve student achievement. That's good news for both charter schools and district-run public schools. And it's very good news for students.

Of course Eduwonkette has a point about the difficulties inherent in drawing conclusions from these types of comparisons - it's difficult to get true random assignment or perfect control groups and there is interference from a host of confounding variables. And it's important that journalists understand and explain these limitations and contextualize the results. But I would argue that there are very different implications and responsibilities when this type of rough comparison is conducted by and reported in a newspaper article than when it comes as a research report from a national and very prominent organization.

It's also important to mention the Post's first story in their charter school series, this one focusing on potential conflicts of interest in the charter school board. Clearly, conflicts of interest are bad and should be avoided, but I'm having a hard time seeing 1) how, exactly, these conflicts of interest manifested themselves in bad decisions by the charter school board and 2) a negative impact on charter schools in D.C., which, as the second story indicates, are doing well in large part because of a rigorous approval process by the charter school board. As this letter to the editor states, it's very important that journalists avoid dragging someone through the mud simply because he happens to work in and have expertise in an area, and then volunteers his time to share that expertise in an official capacity. It sends a message to the business community that they, and their expertise, are not welcome in public education.

Public Goods

The Washington Post deserves praise for the series they've been running recently on charter schools. But this graphic is mislabeled and misleading. Charter schools are public schools too, and it'll be nice when they're seen as complementary, friendly competition to traditional public school systems.

Edubroderism

Like many people, I think President-elect Obama has made a good choice in selecting Chicago Public Schools superintendent Arne Duncan to be the next Secretary of Education. I've seen him speak in public twice and was impressed both times; he comes across as knowledgeable, down-to-earth, and committed to creating better schools for children who desperately need them. While Chicago clearly has a long way to go and the city's NAEP scores still lag other big urban districts, it's been on my mental list of cities that appear to be well-led and moving in the right direction. 

The pick took a while and in the interim a spate of stories appeared characterizing the selection as symbolic of various internecine education policy fights with the Democratic party. Of course, such divisions exist. But there's a growing tendency among various observers to engage in a certain kind of education policy high Broderism, using the disputes as an excuse to call for a renewed effort to build consensus, move beyond entrenched ideological positions, find common ground, set aside anger, and combine the best ideas of both sides in forging a new synthesis on behalf of the children. 

Look, maybe there are education issues where the middle way is best. But maybe there aren't. Sometimes the middle ground is a no-man's-land full of trenches, shell craters, and standing water. As is often the case, opposing ideas are sometimes irreconcilable. The new edu-centrism would be more convincing if people spent a little more time articulating what these synthesis policies actually look like, and why they're better than the ideas currently being debated. That's the difference between a position and a pose. 

More Than Butts in Seats

Education Sector recently completed an extensive process looking at higher education accountability systems in all 50 states, the District of Columbia, and Puerto Rico. In part, we undertook the task to be able to answer comments like this one at the Chronicle of Higher Education:

As a former dean, I was responsible for collecting and reporting “outcome” data on both students and programs to the provost, who then reported it to the appropriate accrediting bodies. I am not aware of any attempt to use the data to inform policy decisions. To use the data in that way would have been completely inappropriate, since we only tended to collect data that was required, easily collectible, and fit neatly into a file for statistical analysis : butts in seats, before and after measures on very elementary standardized exams, student perceptions of faculty, etc. I don’t think any educated person would consider using such data to assess the quality of an institution.

Note how the commenter places contemptuous quotes around "outcomes." While his experience backs up our findings that many places are not collecting enough accountability information, our report documents places where it is being done: The University of Texas System is using Collegiate Learning Assessment and National Survey of Student Engagement scores in a meaningful way. South Dakota is using Collegiate Assessment of Academic Proficiency scores to calculate learning gains for students in their first two years in college. The University of Hawaii-Hilo is using major field exams to test student knowledge across nine disciplines. Ohio is calculating expected graduation rates using student input demographics. A handful of states are using real wage data to track graduates after degree completion. These examples show that it is possible to assess student learning and outcomes across large and diverse higher education systems. States just need to follow these early leaders.

The other thing to note about the above comment is its derision of the current accountability system. Not only was his institution not collecting meaningful data, but they had no mechanism in place to use it effectively. Read our report, Ready to Assemble: A Model State Higher Education System, to see what data states are already collecting and how they are putting it to work.

Do What's Already Being Done

In September 2006, the bipartisan Spellings Commission lamented low college graduation rates, rising student costs, and inadequate information about student learning.

But while the report was correct in its emphasis, it was eventually doomed by the federal government's limited role in higher education. If colleges are going to be held accountable, states will have to carry most of the load. About three-quarters of all undergraduates are educated at public two- and four-year institutions, states provide the bulk of the funding for these institutions, and governors and state legislators appoint the trustees and governing boards that run them. If our colleges and universities are to improve, it must be states that provide the leadership. And, in an economic climate where postsecondary education credentials matter more than ever, it is in the best interest of states to maximize their investments.

In 2008, Education Sector conducted a comprehensive analysis of state higher education accountability systems. We examined thousands of documents, Web sites, laws, and policies for all 50 states, Puerto Rico, and the District of Columbia. We worked to answer two fundamental questions: 1) What kind of information does the state gather about its colleges and students? 2) How does the state use the information it gathers to make colleges and students more successful?

The results were both hopeful and sobering. On the plus side, states are collectively gathering a great deal of valuable information. Some have developed innovative methods to measure student progress in learning, graduation, and success in the work force. Others are carefully tracking the way colleges are distributing financial aid to low-income students. From research output to student engagement to economic impact, states are accumulating more information about more things in higher education than ever before.

But no state is gathering all the information it could. Best practices exist in isolation, with a handful of states tracking important outcomes that most states ignore.

Read today's Education Sector report, "Ready to Assemble: A Model State Higher Education Accountability System," to learn what states are already doing to make higher education systems accountable.

Time for an Education Bailout? California’s Schools Will Likely Need One

The latest numbers from California suggest that the state is running out of money so quickly that it may have to start to pay its bills with IOUs. It is uncertain what the impact of the state’s problem will be on schools, but it looks bad, and is getting worse by the day. In November, the state’s Legislative Analyst estimated a budget gap of around $28 billion between now and June 2010. The annual budget is just over $100 billion with around 40 percent of that going to schools (K-12 and community colleges) (see report here). The budget gap has jumped in the last week to $40 billion, and the urgency is mounting to act fast before the state runs out of money. The Governor has started a debt clock that ticks at $470 for each second of inaction (here) The State Treasurer has suggested that the state may have to stop all construction projects because it will run out of funding paying for its constitutional obligations.

The Governor called a special session in Nov with a lame duck state legislature to address a then smaller gap, and the session ended with no results. He declared a new special session with a new Legislature in December and started with generally the same mix of new revenues and cuts. The political battle is over how much of this gap will be covered with cuts vs. new revenues. The Governor and legislative Democrats (majority party in both houses) are proposing a mixture of new taxes and program cuts, with many differences between the two. In contrast the legislative Republicans are calling for programmatic cuts to solve the problem.

So What Does All of This Mean for Schools?
Schools have been waiting to see how bad the cuts will be. Today’s news suggests that it could be pretty bad. The Senate and Assembly Republican (minority party) weighed in with a proposal that was heavier on the cuts than on the new revenues (here). Combined their plan would address $22 billion of the $40 billion hole. And of that $22 billion roughly half ($10.6 billion) was reduction to K-14 education (K-12 schools and community colleges which are funded together through a constitutional minimum guarantee). In addition they propose significant reduction to early childhood programs. Since this year is already half over, there may not be a lot that schools can do to reduce costs significantly in the current school year, although they better save onto every unobligated nickel. And, while this proposal was quickly blasted by the Governor and the Democrats (here), it is important to recognize that this proposal only addresses half of the problem. So, if the state is going to solve the $40 billion hole, it may take this level of cuts to education or higher plus additional cuts and new taxes.

Why the Rest of the County Should Care.
One in every eight students in the US is educated in California. California current funding per pupil is already below the national average, and near the bottom if adjusted for cost of living. Because of its modest funding and high costs, California schools have smaller staffs than schools in other states – larger class sizes and fewer administrators and other support staff. Take off another $10 billion in funding, and class sizes will balloon even more. At what point does it become a national interest to keep schools from going under. Is it time for the federal government bailout for education? I think this would be a better investment in our future than many of the other bailouts being provided.

Finlandia

We returned from Finland on Saturday, so here are my initial overall impressions, focused mostly on the implications for K–12 education. To begin, let me acknowledge that one can't draw firm conclusions about cause and effect after a short visit. Spending a week in a far-off country means you return knowing a lot more than you knew, and a lot more than most people know back home. You're also armed with various illustrative anecdotes and quotations that are useful to bolster arguments. But I would never claim total knowledge of the American education system, and I live there, spent 19 years in school there, get paid to write and think about it full-time, etc. So my factual assertions will be limited to the obvious (e.g. it's very dark in winter), first-hand observations, and expert sources. When I say, for example, that "Finns are a punctual people," that's based on both experience (e.g. the senior ministry of education official who arrived at an 11:00 AM meeting at precisely 11:00 AM and said "I'm sorry for almost being late.") and official documents (it's a direct quote from the Ministry of Foreign Affairs' official "Guide to Finnish Customs and Manners.")

I'll start by sketching out what Finland is like and how the education system works in broad strokes. It's a remote and sparsely populated nation. There are slightly fewer than 5.5 million people living in a land area about 80 percent the size of California, mostly near the southern coast. The population is racially and religiously homogenous—98 percent are native Finns and 82 percent are Lutheran. For almost 600 years, Finland was under the dominion of Sweden, which is why Swedish is still the second national language and all students are required to learn it in school, despite the fact that the Swedish language minority comprises only five percent of the population. The country's small immigrant population is growing, notably with Russians, Estonians, and Somalis. Finland has very liberal international trade policies, which is more or less a prerequisite for prosperity when you're a long way from everything and your only natural resource is wood. Labor markets, by contrast, are highly regulated, with roughly 70 percent of workers belonging to trade unions, including teachers. The biggest company is Nokia, the cell phone giant.

The Finnish sensibility is an interesting mix of individualism and cultural solidarity. On the one hand, they're very invested in the idea of equality and seem quite comfortable with the high-tax, high-service Nordic welfare state. Because Finland is geographically and linguistically remote—Finnish is a difficult language understood by few non-Finns—they seem to understand the need to stick together. But that mutual support is a means of giving people space to live their lives in an individual, self-directed way. Our hosts at the Finnish embassy in America said that they were far more involved with their neighbors and local community in the U.S. than back home. Finns tend to be taciturn; the chairperson of the Education Committee in Parliament compared Finns to the allegedly indecisive, endlessly voluble Swedes by telling us that "In Finland, we talk a little while, make a decision, and get to work." 

Finland received the highest scores in the world on PISA, an international test of 15-year olds in science, reading and math. That success was repeated on the 2003 and 2006 version of the test. This was, and is, a big deal for them. For most of its history, Finland was ruled by larger, more powerful nations to the east and west. Unlike Americans, they're not prone to think in terms of exceptionality and national greatness.

It's important to understand what Finland's PISA test score distribution looks like beyond the world-beating average. Performance in the top 10 percent of Finnish schools is almost exactly the same as the average among the top 10 percent of all OECD schools. Performance in the bottom 10 percent of Finnish schools, by contrast, is better than the median score for the OECD. In Finland, the Lake Wobegon effect is essentially real—it appears to have few if any low-performing schools. And this is perfectly congruent with the aims of its larger social and economic policies--few people get very rich, but no one is truly poor.

Finnish children don't start 1^st grade until they're seven years old. But most are engaged with state-supported early childhood services from an early age. Parental leave policies are (as Dana Goldstein explains) very generous, and once parents return to work they have the choice of a receiving a child care subsidy or enrolling their children in municipal day care (the most popular option; we visited three such facilities during the week.) They're not in a big hurry to teach reading, focusing more on play and socialization, but it would be inaccurate to describe Finnish day care as non-educational. Half-day "preschool" begins at age six.

All children attend basic primary schools through the ninth grade, when most Finns are 15 years old. All schools follow a single national core curriculum that spells out what subjects must be taught at each grade level, the content to be covered, and the minimum number of hours of instruction. (This includes religious instruction or philosophy for those who opt out.) There are no formal national tests administered to all students a la NCLB. Nor is there a British-style inspectorate system. However, as fellow junketeer Matt Yglesias notes, this doesn't mean that there's no governmental assessment or oversight. National education officials used sample-based assessments to gauge progress, and local municipalities also administer tests as a means of managing their schools. It just happens in a more low-key, non-public way.

Grade retention is virtually unheard of in Finland, homework is generally light, and after-school tutoring is rare. As I wrote earlier, Finns spend significantly less time on education than most countries, particularly the other high-performing nations. While ability grouping is officially disallowed, the principal in the primary school we visited said they try to give more instruction to high-end students in subject like math. While there are no charter schools or vouchers per se, some parents have options among public schools, particularly in Helsinki where population density makes travel to multiple schools feasible. One principal in a school we visited spoke of the school's music and foreign language programs as being key to attracting students. But since standards, funding levels, and teachers in public schools are generally uniform and evenly distributed, and (per above) school-to-school performance variation is unusually low, there seems to be less impetus to create policies designed to engender market competition. 

After ninth grade, the system splits in two. Some students apply to and attend "upper secondary" schools, where they study for three (or sometimes four) years and take college prep-type classes. These students are given a lot of latitude to decide what classes to take (see previous re: independence), and the courses mix students from different age cohorts. Upper secondary students are required to take high-stakes, subject-specific "matriculation exams," the rough equivalent of "A-levels" in the U.K. The results help determine whether students get into the university of their choice—or any university at all. School-level results are publicized by the Finnish media, to the consternation of education officials.

The rest of the students attend three-year vocational high schools, where they receive further education while training for careers. Admission can also be competitive; the vocational school we visited turns away many applicants for it hairdressing program every year. (Hair seems important; one student noted that "Finnish hair is fine and thin, so if your hairdresser makes a mistake the whole village will know.") In one class students were practicing on mannequins while another taught them how to calculate profit margins and otherwise run the financial side of the business. Most Finnish hairdressers are sole proprietors who belong to the hairdressers union. (For those who think welfare states are totally incompatible with capitalism and entrepreneurialism, let me direct you to words such as "profit margins" and "sole proprietors" in the previous sentence.")

The Finnish higher education system has a similar dual structure. There are 20 universities, research institutions built in the classic German mold, and 28 polytechnic institutions where students study subjects like engineering, business and nursing. ("Vocational education" generally has a much broader meaning in Finland than America.) While students can theoretically cross back and forth between the dual tracks, most don't, with the upper secondary schools providing the large majority of undergraduates in both universities and polytechnics. Men are required to spend a year in military service, and it's normal for Finnish students to knock around for a while and not start college until their early or even mid-20s. College tuition is universally free and students also receive a small living stipend while they study.

When asked to reveal the secrets of their PISA success, Finns generally cite two things: egalitarian policies and the quality of the teaching workforce. Finnish teachers are required to get a master's degree from a university in order to get a full-time job. Admission to the programs is extremely competitive, with 10 – 12 percent admission rates overall and a 7 percent rate for the primary teacher education program at the flagship University of Helsinki. A faculty member there told us that applicants came from the top half of the upper secondary pool, which is itself already selective. Teacher applicants sit for a single national exam, with the top scorers moving on to a second screening process based on interviews and in some cases structured teaching observations.

Once they hit the classroom, teachers' salaries are fairly modest, roughly equal those in America. Tenure isn't as automatic as in the states, but all teachers are unionized and enjoy substantial job security. While base salaries are determined by a uniform national schedule, teachers can get paid more to teach in the frozen north or on small islands in the eastern archipelago. Locally-funded performance pay is also an option—in the Helsinki upper secondary school we visited, the municipal government sent the entire faculty on a vacation to Rome as reward for meeting pre-defined (and partially test-based) performance goals. The national student / teacher ratio is slightly below the OECD average, but classes can sometimes be quite large. Teachers are said to enjoy a great deal of autonomy in the classroom—as long as they stick to the national curriculum. "Teachers are told what to teach," one Board of Education official told us, "but not how."

Teaching as an extremely competitive and prestigious profession is obviously quite a contrast to the state of things in the United States. Over the course of the week, we asked almost everyone we spoke with—teachers, principals, ministry officials, politicians—why Finns were so eager to get into teaching. Some cited the satisfactions of professional autonomy. But most came around to some variation of "it's just always been that way." Interestingly, while everyone had clearly thought about this a lot, their historical explanations varied substantially. The consolidated Finnish creation myth of teacher prestige goes something like this:

For many hundreds of years, Finland was a province of neighboring greater powers, first Sweden, then Russia. In the mid-19th century, a new sense of national identity began to emerge, expressed by poets, painters and composers (e.g. Sibelius). At that time, Finland was a very rural society. In every village, there were two important people: the priest and the teacher. Literacy was valued, in part because of Lutheran tradition. So teachers helped Finns become Finns. In the early 20th century the progressive labor movement put a strong emphasis on education and training. Meanwhile, the agrarian movement (now represented by the Centre party in Parliament) put a strong emphasis on the civilization of the rural population. Pro-Christian groups also valued civic education. Many teachers were called to serve as non-commissioned officers in the 1939 Winter War with Russia, a source of national pride. In general, Finnish people understand the vital importance of education to national prosperity and survival, and thus appreciate the role teachers play.

All of which may be true, although as Matt pointed out at one point, many similar things could be said of other European countries where the best and the brightest aren't clamoring to get into the classroom today.

What, then, to conclude about Finland? Despite my recent admonitions, I'm sure that Finnish PISA scores will continue to be deployed as easy evidence in support of various policy agendas. So here are the winners and losers in the "Inappropriately De-contextualized Finnish Education Policy Olympics":

Winners
Teacher unionism
National standards
Mandatory university-based teacher education
Government-sponsored child care and early childhood education
High entry standards into teaching
Teacher autonomy

Losers
Expanded school time
Class size reduction
Strict regulatory and inspectorate-based accountability systems
Increased teacher salaries
School choice

Of course, it makes zero sense to look at things this way. Which is not to say that we have nothing to learn from Finland or other countries; Americans spend too little time considering lessons from abroad. But we have to think about the totality of systems and societies. With that in mind, here's my best guess—and it's a hypothesis, nothing more—about why Finland is so successful and what that means.

In a nutshell, Finland suggests that an egalitarian culture and social policies to match not only make education more effective, they make it less complicated. Or to put it another way: if you know you can trust people, it eliminates the need to do a lot other things.

If you can convince your best students to try and become teachers, for example—even though only 10 percent will be accepted and they'll have to spend five years getting a master's degree—you reap a lot of benefits. Teacher training can be rigorous because the students are smart enough to handle it. Teachers can manage larger classes and work autonomously to achieve common curricular goals. Maybe you don't need to pay them more than a middle class wage (although this is complicated by Finland's very different labor market and compressed range of salaries throughout the economy relative to the American labor free-for-all.) The fact that bad teachers are hard to fire is only a minor annoyance, because there just aren't many bad teachers.

If you provide decent social services and support families with children throughout their lives, then students come to school with fewer behavioral problems, more ready to learn. The high school students we saw were just like ours in many ways—energetic, curious, easily distractable, strangely dressed. But there was an underlying calm to it all that American schools seem to lack. There were no hall monitors, no security guards, and the school administrators reported spending very little time on discipline. The school—and society at large—trusted the students, and the students responded.

All of this makes the primary and secondary schools in Finland good places to work, which makes good people want to work there, which makes them good places to work, and so on. The Finnish combination of social and education policy clearly has many virtues and it's no wonder that many people want to learn from their example. The whole Broader / Bolder agenda essentially boils down to, "If we were Finland, we wouldn't need education reform."

Which may very well be true. But we're not Finland, we haven't been, and we won't be anytime soon. What, then, should we do?

We could start by getting closer. People sometimes dismiss the possibility of learning from Nordic countries out of hand due to their small size and high level of homogeneity. But I don't really buy that. Finland has a lot of empty space, climactic extremes, little arable land or mineral wealth. Nearly everyone is white and the population is dominated by one religion, with most inhabitants living in or near the capital city. But all of those things are also true of Utah; the only difference is that Finland has twice as many people. And the American states that come closest to Finland-level education performance aren't like Utah. They appear to be Massachusetts and Minnesota, both of which have long traditions of liberal policy and only one of which has an obvious Scandinavian cultural tradition. Massachusetts in particular has people from all sorts of racial, ethnic and religious backgrounds. Moreover, there's no inherent contradiction between prosperity and things like generous parental leave, subsidized child care, universal health care and equitable school funding systems. The United States has the 6th highest GDP per capita in the world, while Finland is 20th—but with a lot less poverty. It's not that we can't be more like Finland, it's that many of us just don't seem to want to. 

That said, Americans have distinct national values that differ from other parts of the world, and distinct realities to confront. Our individualism is more rugged, for one. We’re huge and diverse, open to immigration, and changing all the time. Our federal system of government limits the scope of national policies. We don't have the Finnish historical tradition of valuing teachers, wherever it might have come from.

This creates vexing problems of timing and sequence. We didn't do what was needed to create good schools for everyone. But we can't turn back the clock or make ourselves what we're not. There's a fair critique of the contemporary education reform movement that likens it to an escalating series of pharmaceutical interventions—you give someone a drug to solve a problem, and it works to some extent but also creates side effects that require more drugs, and so on with a need for constant monitoring and fine-tuning and escalating complication, all at great expense, when all the while the patient would have been much better off they'd never been sick in the first place. But a lot of our schools are sick, right now. Finland trusts local schools to do a good job (while monitoring performance in a relatively non-intense way), and they respond. Sadly, a lot of American students are educated in municipalities (I live in one) that have historically proven to be untrustworthy.

So, I think we need to move full speed ahead with policies aimed at identifying the lowest performing schools and improving them by whatever means necessary, including shutting them down and educating their students elsewhere, along with creating more public school choices for parents. There's little to learn from Finland here, due to the absence of really terrible Finnish schools.

Finland suggests that you can have national standards without somehow stamping all the individuality out of K–12 education. National standards are seen by many as a political non-starter in the United States, due to the clichéd (but broadly true) observation that conservatives don't like the "national" part and liberals don't like "standards." But that's mostly a political problem. There's really no strong empirical or policy justification for having, say, 51 different sets of standards for 4th grade math, assigned to students based on their residence in political subdivisions that were created via semi-arbitrary historic processes involving essentially non-educational events (i.e. wars, purchase from foreign countries, etc.) People speak from time to time about states as the laboratories of democracy etc. in this area, but that strikes me as mostly nonsensical and really just a way of constructing an after-the-fact policy argument to justify not spending time working on a politically difficult issue.

I'm not ready to endorse the Finnish dual-track secondary / post-secondary system. It has advantages, particularly in the (relative) non-marginalization of students who attend vocational schools and the whole idea of career-focused education. But while the official Finnish education org chart has lots of horizontal lines going back and forth between the tracks, officials there acknowledge that few students actually move from vocational education to university degrees. Putting people in their place so early in life seems, well, un-American.

Finally, it really does all seem to come back to teachers. There's a huge push underway in the K–12 policy world right now to improve the quality of the teaching workforce. But whenever someone suggests doing this by raising some bar or another—e.g. program entry standards, rigor of training programs, certification requirements, on-the-job performance and tenure standards, etc.—the response is always something along the lines of "Where are you going to find all of these new people who want to be teachers? We barely have enough now." Teach for America has already disproved this in principle, at least to an extent. Twenty years ago, graduates of elite colleges weren't clamoring to enter the teaching profession as it was then defined. Then Wendy Kopp came along and defined it differently, appealing to people's sense of service and adding the crucial element of selectivity—and thus, prestige. Teaching in Finland is not a high-prestige profession that anyone can enter. Indeed, there's probably no such thing.

We don't have Sibelius or a compressed wage distribution or a tradition of teacher prestige in America, so we're probably not going to get to a 10 percent program acceptance rate—or the Gladwell / Kane model of letting four candidates give teaching a shot for every one we give a permanent job—anytime soon. But I think we can do a whole lot better than we are. And if we did that—along with common standards and social policies that support families—we could start to break out of the cycle of low performance and increased pressure and political backlash that we're currently in, and move toward a world where education is more trust-driven, less complicated, and more effective all around.

Which Teachers?

All the top consulting, legal, financial, and engineering firms keep a list of schools from which they recruit students each year. They don't attend job fairs at Directional State University, because (supposedly) DSU graduates aren't at the same caliber of those from Harvard, Yale, or other elites. What if you could test it, though? What if we could tell how well each educates their students and prepares them for the workforce?

Louisiana has been quietly doing just that for its graduates of teacher education programs. Starting with mandatory re-designs in 2000-2003, the state now has the capacity to track teacher effectiveness by their educational program. In other words, parents, principals, and policymakers are able to make some informed decisions about which teachers they would want in their classrooms. The most recent review came out this week, and here's the verdict on teachers entering the profession through The New Teacher Project:

The New Teacher Project prepared new teachers whose students, demonstrated achievement in four content areas (i.e., science, mathematics, language arts, and reading) that was comparable or above the growth of achievement demonstrated by children taught by certified professionals who had taught two or more years. Achievement of student learning in one content area (i.e., social studies) was comparable to the growth of achievement of students taught by other new teachers.

Compare that to results like this one:

The University of Louisiana at Lafayette and the Louisiana Resource Center for Educators each had one content area where student achievement was less than that of new teachers. In the content area of language arts, the University of Louisiana at Lafayette program performed at a level where there was evidence that new teachers were less effective than average new teachers but the difference was not statistically significant. In the content area of reading the Louisiana Resource Center for Educators program performed at a level that was statistically significantly less effective than new teachers.

These are very important findings, and they control for student, family, school, and classroom characteristics. The project's next step will be to attempt to answer the why of the results. And, hopefully in the near future, we'll see other states link student growth data with teacher education programs.

Eduwonk's take here. NY Times editorial board here.

Balance

A few years ago, while on vacation in Italy, my wife and I toured a winery in Tuscany and ended up spending an hour or so chatting with the in-house sommelier, a woman in her early thirties. After pouring a really terrific Vino Nobile de Montepulciano, she off-handedly mentioned that she had earned a law degree from a public university but had never practiced, deciding to pursue a career in wine instead. Tuition had been free, so it wasn't an economically hard choice to make. Part of me thought this was great, but another part started worrying about subsidy-induced overconsumption and the fact that some other student had been denied the chance for an expensive education that she had essentially wasted. Then I had another glass of wine and stopped worrying about it, because Tuscany is, in fact, just as nice as they say.  

College tuition in Finland is also free, even for non-EU foreigners. (Memo to students: if your college charges you full tuition to attend the University of Helsinki when you take a semester abroad, they're ripping you off.) This is widely seen as a bedrock principle and is unlikely to change anytime soon, at least for Finnish (and thus EU) citizens. But Finland also has a recently-established system of polytechnic institutions that are eager to grow in stature and compete with the more prestigous, long-established research universities. The national legislature is currently considering an ambitious shake-up of the whole postsecondary system that would give the institutions more license to raise private money and otherwise act in a more autonomous, self-interested way--to be more like American institutions, in other words. At some point, the tuition question will likely end up on the table.

From a policy perspective, there are basically three options: no tuition, market-rate tuition, or somewhere in between. Pure market rate tuition keeps low- and middle-income students out of college and reduces overall educational attainment. No tuition solves that problem but has side effects like legal expertise slowly dissipating in the Tuscan sun, as well as Finnish humanities students taking epic amounts of time to complete their degrees. It also makes universities wholly dependant on the economic fortunes of the state and the whims of politicians for funding.

The best solution, in theory, is somewhere in between--highly subsidized but not non-existent tuition, so students have a stake in their education and universities have multiple sources of revenue. The problem is that of three options, this is by far the least stable, because the various actors involved have fundamentally different interests, and public tuition policy becomes a 24/7 arena in which those interests constantly collide. Students, like all consumers, want the best possible value. Public policymakers want universities to produce the maximum possible number of well-educated graduates using some finite level of resources. Universities want as much money as they can get from everyone. From the standpoint of pure rational self-interest, it always makes sense for a publicly-subsidized university to raise tuition, as long as they keep it somewhere below the market rate. Since the distance between the subsidized rate and the market rate is generally quite large, there's a lot of potential for change--and thus, things to argue about. 

The only way to manage this is to maintain some degree of shared values among all three parties whereby everyone recognizes--and on some level, accedes to--the interests of others. This is easier to do in a country like Finland, a nation of 5.5 million people who are unusually homogenous and committed to egalitarian ideals. America is a more unruly and complicated place. Diversity and self-interest have given us a tremendously robust and successful higher education system (at least at the top end). But it also makes it hard to agree on important things. 

The Big Picture

One of the benefits of spending a whole week doing nothing but learn about a single foreign education system is that it forces you to consider the totality of things in a way that's actually very difficult in one's home environment. For example, I spend very little energy wondering how America's schools could be improved if we implemented a financing system whereby the federal government provides 80 percent of school resources, rather than the 10 percent it actually provides, because the odds of such a policy coming to exist in my lifetime are very low. That's just not how we roll in the United States. But of course the basic finance structure does matter, a lot, and--crucially--affects how much other things matter. Every piece of the system is contingent on other pieces and the overall design. 

This also underscores the absurdity of education policy arguments that go something like this: Country A is kicking our tail on some agreed-upon measure of achievement. Country A has Policy X, which is very different than our policy. Therefore, implementation of Policy X here in America will improve achievement. People say stuff like this all the time, and their arguments are generally given a lot of weight.

But they shouldn't be, at least not if they're presented in such in simplistic way. Take, for example, the issue of school time. There's a growing movement in America to invest a lot of resources in expanding the school day and otherwise increasing the amount of time students are educated. Inevitably, these discussions come around to the fact that countries like Korea and Japan have much longer school years than do we, provide all kinds of after-school tutoring, and generally do much than we do on international tests, particularly in math. Malcolm Gladwell made a version of this argument in his recent book and the school time people trot it out at every opportunity. It's one of those little nuggets of conventional education policy wisdom that everyone knows.

Yet two days ago I sat in a conference room at the Finnish National Board of Education and listened to an education official explain that Finland, which also kicks our tail in math, spends less time on math instruction, both in school and out of school, than does Japan, Korea, other Scandinavian countries, and the OECD average. And Finland has the highest math scores in the world. She didn't present this fact as a puzzle; she offered it as evidence of why Finland does so well. "Learning is efficient in Finland," she said, and this was her proof.

Does this mean that the school time movement is a fraud? Of course not. Long school days and years may indeed be good for Japan, given the nature of Japan, the Japanese, and the rest of the Japanese education system. It may be a good idea in the United States, or some parts of it, given who we are and all the other things we do. Or it may not; there's no way to know without understanding all of the parts and how they fit together. More on those parts and the big picture later this week and next.     

Duncan's Data

On Monday Alexander Russo asked for more information on how Chicago Public Schools have fared under Superintendent Arne Duncan, a likely Secretary of Education candidate. Eduwonkette gave a harsh review of the data, but the truth is a little more mixed.

Since Duncan took over in 2001, Chicago has made statistically significant progress in fourth and eighth grade math and fourth grade reading scores. They're up across all subjects and grades for low-income students, students with disabilities, and English Language Learners (ELL). Low-income students narrowed achievement gaps in all but fourth grade math, while students enrolled in special education and ELL students closed gaps in both eighth grade subjects.

To Eduwonkette's point, the racial achievement gaps have not narrowed as much as we'd like, but blacks are scoring higher in 3/4 categories and Hispanics on all four.

Do these data cement Duncan's candidacy or disqualify it? Neither, really, but probably more the former than the latter.

Another Californian Unemployed

California recently announced that over 26,000 had lost their jobs in the month of October. Its unemployment rate has risen to 8.2 percent, one of the highest levels in the country (only Michigan and Rhode Island are worse off). Yesterday one more fell victim to this trend. David Brewer the Superintendent of Los Angeles Unified, the countries second largest school district, was asked to step down after only two years on the job (LA Time Article). Of course he is better off than the other 26,000 newly unemployed in that he will be leaving with a $517,500 exit package. In fact, to pay for his severance package, it is likely that the district will have find several other educators will join him in the ranks of the unemployed as the district continues to face large budget deficits. Of course maybe he could fill one of the vacancies on the Schwarzenegger’s administration which has had difficulty keeping education advisors. The Governor has had four Secretaries of Education in 5 years. Currently, the Governor’s education advisory team, the Secretary of Education’s Office, is almost empty with the following openings – Secretary of Education, undersecretary, chief of staff, and K-12 assistant secretary.

Tier Ducks

The latest international test results are in, and they bring mostly good news for US educators. Yesterday's release was the fourth edition of the Trends in International Mathematics and Science Study (TIMSS) since the original 1995 administration. Rather than use TIMSS merely for hyperbole, it's worthwhile to look at them more holistically.

First, TIMSS should not be used merely for rankings. While it's technically accurate to say the US had the ninth highest score in 8th grade math, for example, just that number alone does not do justice to the truth. Five countries (Chinese Taipei, the Republic of Korea, Singapore, Hong Kong, and Japan) scored significantly above us, five countries scored about where we did (Hungary, England, Russia, Lithuania, and the Czech Republic), and 37 countries scored well below us (including places like Australia, Sweden, and Norway). Similar tiers exist across fourth and eighth grades for math and science.

The results are meant to show interesting across-time comparisons as well, and in that respect, we're doing quite well. Our scores have risen both in raw numbers and against the average. At the same time, we've also narrowed gaps in mathematics since 1995 for blacks and whites, whites and Hispanics, and low- and high-achievers:

• 4th grade white-black gap fell from 84 to 67
• 4th grade white-Hispanic gap fell from 48 to 46
• 8th grade white-black gap fell from 97 to 76
• 8th grade white-Hispanic gap fell from 73 to 58
  

Despite this progress, the biggest difference in the scores of US students is not between countries, but rather remains within our own. In fourth grade math, the effect size of US students attending high-income versus low-income schools is 1.4 times as large as the difference between US students and the highest performing country. In science, the effect size by income is three times what it is between the US and the leading nation. Income gaps continue to persist at levels higher than all others, and that should be the real story out of these results.

Edu-Jobs

If you're looking for a job in the education world, this might be a good place to start. It's the Public Charter Schools Job Board, and it lists openings nationwide for everything from teachers and principals to assessment specialists and chief financial officers.

Finland Cont'd.

As you'd expect, Finland's child care policies are more generous than ours; Matt Yglesias explains more here and here. Meanwhile, on the teaching front, all K-12 teachers are required to go through rigorous university-based training, in most cases through a master's degree. But only 10 - 12% of applicants to university teaching programs are accepted. In other words, the system seems to be roughly what you'd get if you locked Linda Darling-Hammond and Wendy Kopp in a room and didn't let them out until they'd struck a grand bargain about the nature of teacher selection and training. This raises some interesting path dependency-type questions about education; viz. the extent to which various generally admired aspects of the Finnish education system are contingent on the caliber of its teaching workforce and the resulting implications, or lack thereof, for American policy.

We're moving our way through the Finnish education system chronologically, starting with early childcare centers on Monday, lower secondary yesterday, upper secondary today, and higher education later in the week. The school we visited yesterday is in the part of town where many recent immigrants live and thus belies Finland's reputation for total racial / ethnic homogeneity. The biggest immigrant populations, we are told, come from Russia, Estonia, and Somalia. Oh, and Iraqi refugees of course, which is just wonderful to contemplate as an American.  

Meanwhile, I won 300 euros playing blackjack in the Helsinki Grand Casino last night. Gambling is awesome! Seriously, I don't know why people don't do it more often. 

Equality

Nordic countries are famous for their egalitarian attitudes and social policies, and so far the Finns have said nothing to contradict this. In the U.S., there's significant variance in funding between school districts. (Contrary to popular wisdom, this is not primarily a function of differences in local property wealth--state funding passed local funding as the single largest source of school revenue in the 1970s and roughly two-thirds of inter-district funding variance nationwide is a function of wealth differences between states, not differences within states. The richest states have roughly twice the taxable wealth per-student compared to the poorest, and this matches inter-state spending differences almost exactly.)  In Finland, we are told, funding is centralized and equitable; this morning the principal of a school we were visiting claimed that "It doesn't matter if you're rich or poor, you go to the same school." This kind of equality is a function of a lot more than funding, of course, extending also to a national curricula and an unusually high-quality pool of well-educated teachers. 

This can create conflict with the natural desire of parents to provide more to their children, as with the case of a famous Formula One driver (last week we were told that Finns specialized in certain occupations, including race car drivers, ice hockey goalies, and international diplomats) who wanted to pay for an extra teacher specifically to benefit his son, who attended the school we visited. The principal said no, that wouldn't be possible, causing the driver to rant about this $%^&ing communist system, etc. 

Now he's supposedly planning to come to America and drive for NASCAR. So there may be some self-selection at work here as well. 

Convicted

Earlier this year I wrote a long magazine story about four African-American drug dealers from West Baltimore--essentially, the real-life guys from The Wire--who, after being arrested and indicted on a raft of federal murder, drugs, weapons, and conspiracy charges, suddenly began using a bizzare constitutional fundamentalist defense that has origins among white supremacist militia crazy people. When the story was published, they hadn't actually gone to trial yet; I just assumed they would be convicted since their defense amounts to legal gibberish. Yesterday that's exactly what happened; all four were convicted on most of the charges, three of them for first-degree murder. 

The Ryan Leaf Syndrome

What do leaders in other industries do when they aren't certain about which credentials matter for success? If the markets for football quarterbacks and financial advisers are any indication, as Malcolm Gladwell argues they should be, education leaders must be willing to interview and try out many candidates for teaching jobs.

Making professional sports predictions based on collegiate (and sometimes high school) success is notoriously difficult. Gladwell uses the especially difficult case of NFL quarterbacks to make his point: NFL teams, despite watching hours of live game films and testing players' speed, agility, strength, and intelligence, still are pretty bad at predicting which quarterbacks will eventually be successful in the pros. They make horrible decisions based on their faulty decision-making that have real consequences for their franchise.

Take the case of Ryan Leaf, a wildly successful college quarterback for the Washington State Cougars. The San Diego Chargers traded three draft picks, a reserve linebacker, and a Pro-Bowl running back to move up one space, from second to third, just to have the right to draft Leaf in 1998. The Indianapolis Colts were drafting #1, and there was serious debate about whether they should opt for Leaf or the other top quarterback in the draft, a guy by the name of Peyton Manning. The Colts took Manning, the Chargers Leaf, and one became a household name. But before we knew which one it would be, the Chargers believed they had a franchise quarterback. They lavished Leaf with a four-year, $31 million contract and an $11 million signing bonus. Before beginning his first season, Leaf said he was, "looking forward to a 15 year career, a couple of trips to the Super Bowl, and a parade through downtown San Diego."

Leaf started the 1999 season well, becoming only the third rookie quarterback in history to lead his team to a 2-0 (preseason) start. But by his third game, Leaf managed to complete just one of his 15 passes. It went for only four yards, and he fumbled three times. He was benched after nine games in which he threw 13 interceptions compared to only two touchdowns. After bouncing around the league, Leaf was forced to retire in disgrace at the age of 26.

This is what happens when an organization bases their personnel decisions completely on what happened in the past. Leaf had all the tools--he threw the ball with speed and precision, had succeeded in major college sports, and had the body to withstand the demands of the National Football League--but he couldn't cut it, and his team suffered the consequences.

Contract that experience with how financial advisers are recruited. As anyone who's seen The Pursuit of Happyness or read Gladwell's piece understands, financial advisers have few input requirements. Instead, they're chosen through a highly competitive process. A wide open field of candidates--the guy highlighted in Gladwell's article regularly interviews at least 20 candidates per job opening--are whittled down based on work habits, not necessarily on their education credentials. Then, after an intensive review period, they are given an apprentice role. After another three or four years, the firm is finally in possession of what it considers high-quality financial advisers. This is a lengthy and expensive process, and it requires extensive human capital development and a comprehensive system for sorting high- and low-achievers.

If we apply these lessons to human capital development in education, we start to understand that the traditional teacher professionalization model is backward. We shouldn't even try to sort out the difference between the Ryan Leafs and the Peyton Mannings of the teaching world. Instead, we should break down barriers to entry, encourage high-quality applicants from diverse backgrounds, and use some initial criteria to sort applicants based on work ethic and intelligence, but then focus most of of our attention on what happens on the job. Until school districts have the data and political courage to do so, they'll be like NFL teams choosing between Tim Tebow, Sam Bradford, Colt McCoy, Graham Harrell, or Chase Daniel. They're all fine, but which one will be great and which one will be Ryan Leaf?

Reason to Worry

The Chronicle of Higher Education's Paul Basken reports today that student loan providers are worrying about the economic health of colleges. Private colleges that don't have a large endowment and are heavily dependent on tuition for revenue could face problems balancing their budgets in light of the recent tightening in the private student loan market. Students are having a harder time taking out the private student loans needed to pay tuition, meaning that they are likely to move to a lower cost public college, a 2-year institution, or just stop attending altogether. And if small private colleges can't meet their enrollment goals, they may be facing a serious financial problem.

But this situation isn't just the result of another credit crunch in need of a bailout. In fact, private loan debt was too easy to get for a while, meaning that many students who could not afford large, high-priced student loans were getting them anyway. In that sense, the recent restriction of private lending is a correction to the market (much like the reduction in subprime home loans). And, during the time of easy credit, many colleges engaged in some convoluted tuition pricing - raising the sticker price while providing lots of merit aid to recruit high scoring students.

Basken quotes Daniel Meyers, president of First Marblehead, a leading private loan company, as saying that "colleges are experiencing 'this very strange effect' where more than 1,000 institutions are 'all trying to charge $45,000 or $46,000 a year, most very unsuccessfully...and consumers have woken up to be much more discerning people'." More discerning consumers are not a bad thing - before the credit crunch, many students were able to take on a lot of debt for a degree that wasn't worth the price tag.

When we return to more normal credit markets, hopefully both lenders and students will have become more discerning consumers, with lenders doing a better job of assessing student risk factors and the value of degrees, and with students thinking critically about whether a particular college is really worth a $45,000 price tag.

Helsinki...

...is very dark this time of year. I knew this was true in theory, but having never ventured this far north in the winter before, I wasn't quite prepared for arriving at 3:30 PM, i.e. dusk, walking around the city at 4:30 PM, which might as well be 4:30 AM in terms of darkness, having dinner, going to bed, sleeping for eight hours, waking up, having breakfast, getting into a van, arriving at a day care facility for our scheduled tour, and it's still dark. It's a Fiddler on the Roof light pattern, basically--sunrise, sunset, all in the space of about six hours. That said, the people are friendly and as is always the case when you travel abroad, one's sense of possibility as to how things in America might be different is expanded, in terms of everything from public policy and cultural attitudes to food and bathroom fixtures.   

Finland Bound

I'm leaving tomorrow for a week-long junket fact-finding mission to Finland along with some other think tank people and journalists who will be learning why the Finns are beating the world on PISA and other measures of education success. We'll be in Helsinki, so if you have any suggestions about places to go and people to see--education-related or otherwise--send me an email, kcarey at educationsector.org. And if you actually live in Helsinki, we're staying in the Sokos Hotel Vaakuna Helsinki--drinks are on me. 

One of the nice things about vacationing in visiting Finland in an official capacity is that you get invited to the Finnish embassy here in DC for a pre-trip orientation / dinner / sauna bath. The embassy is just as you'd expect, all blond wood, glass, steel, and elegant Nordic design. The dinner included meatballs and I know just enough about Finnish history not to ask if they were Swedish. Afterwards, we retired to the embassy's extensive subterranean sauna facility--the only "diplomatic sauna" in Washington, FYI--where our host, a cheerful broadcast journalist, regaled us with stories of his youth spent in a special army ranger outfit based near the Arctic Circle whose mission was to travel via cross-country skis to positions on the Eastern border and repel attacking Soviet helicopters with shoulder-fired missiles. Why this hasn't replaced the comparatively-much-less-exciting biathlon as an official Winter Olympics sport is a mystery. 

Earlier in the evening, he suggested that I should ask Finnish education officials why, if the Finns rank so high on PISA, the University of Helsinki is only the 68th best university in the world. But while I hope to learn about many things next week--early childhood education, recruiting high-quality teachers, and the Bologna process, among others--that one I already know. 

In primary and secondary education, success is defined in terms of how much students learn. We can--and do-contest the hows and whys of it, but no one really debates the principle. In higher education, by contrast, success is defined in ways that have little or nothing to do with student learning. The world rankings he cited--which place 17 American universities among the top 20--are based entirely on research measures like publications, citations and prizes. A university could literally not enroll undergraduates and it wouldn't affect their position a bit. 

OECD is now in the process of piloting a higher education version of PISA. Countries that lack world-renowned research institutions may find their standing improved when the focus shifts to student learning. But Americans won't know if our unchallenged reputation for having the world's best higher education system is deserved, because we have--sadly--chosen not to participate. I guess when one measure says you're the best, there's no reason to support any others. 

Are Value-Added Effectiveness Measures Good Enough to Use for Compensation Decisions?

There’s a great deal of attention being given to using test scores to measure teacher performance these days, recent announcements from the Gates foundation ensure this will be high on the national agenda in coming years. But recent studies show that the value-added measures contain significant amount of error. Which raises questions: how can imperfect measures be incorporated into high-stakes decisions like teacher pay? How good is good enough?

Reformers have been waiting for longitudinal data systems to be implemented to provide value-added data to support improvements to compensation and retention decisions. The data is now there in several states, but the quality of the new information may not be as good as many of us had hoped. Just before Thanksgiving, two new studies were released that show the lack of stability of value-added measures of teacher effectiveness over time. The first by Dan Goldhaber looks at North Carolina data to see if pre-tenure teacher effectiveness (measured by the value-added gain of a teacher’s students) is a good predictor of their effectiveness post-tenure (here). The study showed that a teacher ranked in the bottom quintile of teacher effectiveness has a 32 percent chance of being in the bottom quintile post-tenure. While this is better than random (random would be around 20 percent), it is not much better than random. At the same time, 11 percent of the poor performers pre-tenure (bottom quintile) end up being in the top quintile post-tenure. The measure is a little more consistent at identifying top performers – 46 percent of top performers are top performers post-tenure (see Table 1 for all measures of pre and post value-added effectiveness). Goldhaber also looked at using the first 3-years of data to predict outcomes, and the predictive power does not change much.

Table_1a.pdf

A second paper by Tim Sass shows similar results from California and Florida studies. (here)
This paper focuses on whether value added measures of teacher quality are stable enough to use for compensation decisions. It shows similar results as the Goldhaber study over time. The lack of stability over time may not be surprising given the group of students a teacher gets each year is random. The data can not measure whether a teacher had a particularly disruptive class in the first year, and a better group of students the next. So, the randomness of classroom make-up may have a lot to do with these results. The Sass study shows that while measurable, student characteristics explain some of the differences in value-added effectiveness, but most of the differences across time are unexplained (See Table 2 for complete effectiveness measures).

figure_2.pdf

The part of the Sass study that caused me the greatest concern was how inconsistent these value-added measures are across tests. Students in Florida take two tests annually. They take a low stakes norm-referenced test and a high stakes standard-aligned test. Sass looks at how stable these value-added measures are across these two tests. So for this comparison, the random draw of students is the same for any given teacher. While these results look a little more stable (43% of bottom quintile teachers remain in the bottom quintile on the other exam), they are not as stable as you would hope. If just switching the exam moves 5 percent of teachers from the bottom of the distribution to the top, it would likely make teachers question the validity of the measure reflecting true effectiveness.

These papers and a few others suggest that value-added measures are not very consistent over time, and may not be the panacea for which some reformer have been hoping.

How Good is Good Enough? Now you would think that the bar for improving teacher compensation and tenure decisions would be pretty low. The current compensation structure is based almost exclusively on a teacher’s years of experience and college credits/advance degrees. Advance degrees have been consistently shown to have no impact of teacher effectiveness. For experience, teachers appear to improve their craft slightly over the first two to three years, but additional experience does not seem to have any impact. Clearly moving to value-added compensation could more accurately reward effective teachers than the current system. However, if a compensation system were based partially on these value added measure, I think that teachers would perceive the outcomes above as too arbitrary. It also makes me think that principals and mentor teachers could do a better job of predicting effectiveness than last year’s test results. (See Brian Jacobs on this question – principals seem to do pretty well on the identifying teachers at the top and bottom of the distribution, but their measurement is less predictive than prior year’s value-added (here). Of course this is not an either or choice. Can principals armed with value-added test results do an even better job than either one alone? What about a combination of principal evaluations, mentor teacher evaluations and value-added? Are there more rigorous evaluation methods like those of the Teacher Advancement Program or others better predictors than the value-added measures? (See Ed Sector Report on Teacher Evaluation here) As with all good research, it leads to more research. And with Gates interested in these topics, more research is likely to be on its way.

Two Steps Back

Two pieces of bad news today for those working to build a quality supply of public schools:

First, the 1st District Court of Appeals in Florida ruled that a 2006 law creating the Florida Schools of Excellence Commission conflicts with the Florida state constitution. According to the court, the Commission, which would be an independent, statewide office established to approve and support charter schools, was unconstitutional because the Florida constitution limits oversight of charter schools to local school districts.

Well, it looks like it's time to change the Florida constitution. A growing body of research shows that having, in addition to local school districts, one or more professional authorizers whose sole focus is approving and overseeing charter schools makes for a healthier and higher quality charter school sector. In fact, a 2006 ES report on Florida charter schools stated that the proposed Schools of Excellence Commission, "will likely reduce the number of appeals to the State Board and relieve unwilling sponsors of their chartering responsibilities while significantly improving the quality and transparency of authorizing across the state."

And the second piece of bad news is the bailout bill passed in the House Education Committee in Michigan which allowed Detroit to keep it's "first class school district" status (reducing the enrollment threshold from 100,000 students to 60,000 students) and thereby limiting the opening of new charter schools in the district. As we've reported, maintenance is required among Michigan's charter schools, but a blunt limit on opening new charter schools does nothing to improve the quality of charter schools or the quality of Detroit Public Schools.

Darling-Hammond Unbound

Score one for the KAPPAN magazine. The edu-magazine has a very timely piece on school accountability in its just-mailed December issue by top Obama policy advisor Linda Darling-Hammond. Anonymous "reformers," some of whom also have ties to the Obama administration-in-waiting, have been taking shots at the Stanford professor during the transition, in part because she has been tough on the quality of state testing under NCLB. They have declared her to be "anti-accountability."

The KAPPAN piece provides a valuable window on her thinking. She's indeed not a fan of NCLB-brand multiple-choice testing. "NCLB reinforced using test-based accountability to raise achievement, yet the US has fallen further behind on international assessments of student learing since the law was passed in 2001," she declares at the top of the article.

Darling-Hammond has spent a lot of time studying the teaching and testing systems of high achieving industrialized countries and likes them better than ours. Among other things, she says, they teach fewer topics in greater depth; focus more on reasoning skills and applications of knowledge rather than on coverage of content; and rely heavily on open-ended questions "that require students to analyze, apply knowledge, and write extensively," in contrast to US tests that "rely primarily on multiple-choice items that evalute recall and recognition of discreet facts." She's right about that.

Darling-Hammond points approvingly to a "growing emphasis" in high-performing countries on "project-based, inquiry-oriented learning" that has led "to an increasing prominence for school-based tasks, which include research projects, science investigations, development of products and reports or presentations about these efforts"--so-called performance tests. The bulk of the article (written with co-author Laura McClosky) describes approvingly locally administered peformance assessment in countries ranging from Finland to Australia, Hong Kong, Sweden, and the UK.

There's little doubt that LDH would push to introduce these kinds of assessments into US public education if she were to have a senior role in the Obama administration: We need, she writes, "a new vision of assessment" in American education. To the extent that testing drives teaching, that would be a good thing.

The question is whether she would push to incorporate performance tests into NCLB-style statewide testing systems, or try to move testing down to the local level.

In writing that "the policy community has little understanding about how systems of assessment for learning might be constructed and managed at scale," she is acknowledging the challenges of using performance testing under the NCLB system, cost and scoring reliability chief among them. One thing she might do if she goes to work for Obama would be to have the federal government sponsor an effort to address the difficulties of doing performance testing at scale, and give states financial incentives for using such tests. Given the growing consensus that well-crafted performance assessments would represent a big step towards teaching students the higher-order thinking skills that they need today (Darling-Hammond points out that US students score lower on problem-solving that their international counterparts), this would be a smart investment--and a refreshing change from the Bush administration's hear-no-evil, see-no-evil stance on test quality.

Another possible policy solution, she implies elsewhere in the article, would be to include performance-based local assessments into "overall examination scoring systems." That's what several of the countries she has studied do.

The big question for supporters of NCLB's statewide standardized testing systems is whether performance assessments would be used for holding educators accountable for student achievement. Much edu-blood was spilled over that question a year ago, when Rep. George Miller included the notion of local assessments in a draft NCLB reauthorization bill.The combatants eventually withdrew from the field and the Miller draft was decommissioned.

But it's clear that Darling-Hammond is ambivalent about using performance testing to hold educators accountable for student achievment. She notes that the countries she has studied "do not use their examination systems to rank or punish schools or to deny diplomas to students." Finland, she writes, "has no external standardized tests to rank students or schools." Instead, she writes approvingly, the testing systems in Finland and other countries are closely linked to efforts to develop teachers' ability to teach higher-level skills to their students; they are part of the countries' human capital strategies.

So, if Barack Obama gives Linda Darling Hammond a major role in his administration, we're going to have a big policy debate over testing in American education and whether we should move beyond NCLB accountability to something potentially very different. Such a debate wouldn't be a bad thing.