Comparing Treatments

The recently-passed stimulus bill provides money for comparative analysis of medical treatments for various ailments. It's the first such authorization, and it will allow us to answer whether ailment X is best treated with pills, therapy, or surgery. These types of comparisons have long been absent in discussions of educational pedagogy, but yesterday's IES/ Mathematica report does just that. It looks at four common math programs that collectively control about 32 percent of the K-2 math curricula market. It found statistically significant scores for students using two of the four programs:

average math achievement of Math Expressions and Saxon students was 0.30 standard deviations higher than Investigations students, and 0.24 standard deviations higher than SFAW [Scott Foresman-Addison Wesley] students. For a student at the 50th percentile in math achievement, these effect sizes mean that the student’s percentile rank would be 9 to 12 points higher if the school used Math Expressions or Saxon, instead of Investigations or SFAW.

One program (Saxon) offered one additional hour per week of instruction, which suggests its success may owe partially to additional time expenditures, but the two successful programs tended to offer more lessons per week devoted to word problems, addition and subtraction of facts with whole numbers, money, place value with whole numbers, fractions, probability, decimals, and percents.

The study looked at 39 schools implementing new math curricula in the 2006-7 school year. Researchers added 71 additional schools for 2007-8, so we'll have expanded results next year. Math is a good place to begin this comparative process, though, because for too long we've relied on industry-created demonstrations of effectiveness. While this study is only preliminary, it's extraordinarily useful to have objective, comparative results on educational programs.