What’s So Scary about a Survey?

The 2008 Brown Center report, authored by Tom Loveless, takes a close look at PISA – the math, reading and science test delivered to 15-year-olds by the OECD – and raises five objections to the test’s methodology and implementation.

The 2006 PISA tests more than 400,000 students from 57 countries representing 90% of the world economy. PISA differs from its main international competitor, the TIMSS math and science test administered by the US Department of Education, in its focus on policy recommendations and emphasis on the application of knowledge to novel problems. Out of the 30 OECD countries tested in PISA, the United States ranked 21st in science and 25th in mathematics (a printing error led to unusable results in reading).

Loveless raises five objections to the PISA test. First, he notes the lack of nongovernmental participation in the test’s policy recommendation process. Second, he criticizes the test’s alignment as overly broad for assessing the effectiveness of schools in particular. Third, he argues that PISA analyses selectively choose variables used in scatter plots, distorting the nature of the relationship between variables. Fourth, he suggests that PISA’s policy recommendations are disconnected from the assessment data. Loveless’ strongest critique, however, is reserved for a series of survey questions in the test which gather students’ self-assessment of their awareness of issues and attitude towards public problems.

PISA asks test-takers questions about their “values, motivational orientations and sense of self-efficacy.” It’s not hard to understand why a student’s career motivations might be closely tied to their abilities. PISA finds, not surprisingly, that a student’s success in science is moderately related to whether the student finds science useful and scientific problems worth solving. Loveless objects, however, to questions on values: students who disagree with government protection of endangered species or regulations on factory emissions are labeled as lacking a sense of environmental responsibility by the test. “It is difficult to see how declaring support or opposition to a policy without knowing the details is a sign of responsible citizenship,” Loveless writes. “It may in fact be exactly the opposite.”

Citizenship is not necessarily ideologically neutral. PISA’s responsibility questions are part of a broader effort to understand issues of student motivation and attitudes which may shed light on differences between nations. PISA doesn’t just measure schooling in math, reading and science, it measures literacy. True literacy, at least in part, means students who can and will use their education as a form of economic mobility and social empowerment. At the very least, PISA’s questions are another attempt to understand what motivates students to do well and what social factors line up with good educational outcomes. What’s so wrong with a little more data?

By Chase Nordengren.