The (preliminary) research by Roland Fryer and Will Dobbie attempted to separate any gains (or losses) due to HCZ activities (it provides free clinics to expectant mothers, after-school programs, a college-success office, community health programs, and so on) from accomplishments attributable to HCZ's Promise Academy schools alone. It's not easy to separate these things, but Fryer an
d Dobbie took special care to do so. They compared HCZ children not just to city or state averages, but took advantage of the oversubscription of the schools (more devoted parents wanted to enroll their kids in the schools than there were available seats) in order to compare lottery winners to lottery losers. In addition, they compared students who attended Promise Academy to siblings who did not (because the parents of such children received the same support services, the researchers were able to separate the effects of the services and the effects of the schools combined with the services).The chart at left shows how the 2005 cohort of entering 5th graders have fared over time. Losers of the lottery actually had higher fourth and fifth grade scores on the state math exam, but Promise Academy students have erased the small gap by 6th grade, opened a lead on their peers in 7th, and are well above by 8th grade. They have not quite closed the black-white achievement gap, but they have narrowed it significantly.
These results come after controlling for gender and income. This is key, and one of the biggest things critics of the study have missed. Pallas attempts to show the report's errors by presenting Promise Academy's scores on state math and English tests for the same cohort. His graph shows only modest gains in gap narrowing, but it's because he doesn't control for anything. It might seem insignificant to only close the black-white gap and not the income gap, but the charts clearly show the former to be sizable and persistent.
Pallas also argues that the results are invalid because they come on the high-stakes but less-valid state tests. That may be, but his own bias comes out in the comments when he writes that,
In other words, he criticizes the HCZ for not showing their effectiveness on "good" standardized tests, but even if they had, it wouldn't be enough because test scores are bad.
As a general rule, I think it’s a bad idea to rely on a single assessment to make judgments about the efficacy of schools, or educational programs or policies. We expect public schools to contribute in so many ways to the social and intellectual development of children and youth, and no single measure can come close to capturing the full spectrum of goals that we have for our public schools.
Moreover, Pallas says, Promise Academy children don't score nearly as well on more rigorous exams. His alternate data comes from low-stakes exams that students might not work hard on, but this criticism would still work if we saw evidence of some sort of ceiling effect. If Promise Academy did a really good job of raising kids just to a bare minimum, just above the "proficient" line on state tests, the school's scores would jump but the kids wouldn't have learned much.
This chart shows
that's not happening. The entire (blue) curve of Promise Academy students shifts to the right over time, indicating that all are making progress, not just the "bubble kids." The red line is the lottery losers, and it shows that Promise Academy children advanced at much faster rates than children from other motivated families who entered the lottery and lost. The results were not just limited to math, either. Promise Academy children made statistically significant gains in English Language Arts compared to their peers, albeit smaller ones than in math.These findings are all well and good, but how do we disentangle the effects of schooling from the effects of everything else? Fortunately, there's a natural experiment already occurring. The lottery winners and losers, both made up of HCZ students, show that there's something additional gained by attending the Promise Academy. Also, Promise Academy parents have access to all of the programs mentioned above plus nutritious food, pre-made meals, travel vouchers, and general advice about supporting their children in school. These parents have children who are enrolled in Promise Academy and those who are not. The parents and families are the targets of these programs, which means their benefits should accrue relatively evenly among their children. They don't. The siblings of Promise Academy students do achieve slightly higher than their peers and miss fewer days of school, but these effects are nowhere near the ones observed in the Promise Academy students. There must be something about the combination of services and schooling to account for such differences.
David Brooks has a political agenda and only 750 words to write about it, so he takes these findings and runs with them. He sees the school as the one extra element and takes that to mean that the school is what made the difference. He might be right, but in the process he ignores the possibility that the combination of intense services and intense schooling made the difference.
It's fair to criticize Brooks for coming to the wrong conclusion. What isn't fair is to attack Promise Academy or the study itself.
8 comments:
Excellent post, Chad. But technically, Brooks conclusion isn't wrong. The school is one of the important variables. But as you point out, it's only one variable. He could have, and probably should have, acknowledged the other factors that led to the successful outcomes, which would have made his conclusion more accurate.
Chad,
As might be surmised from a post entitled “Just How Gullible is David Brooks?” my critique was focused on Brooks’ claim that the HCZ Promise Academy had eliminated the black-white achievement gap. The comparison of lottery winners and losers in Fryer and Dobbie’s paper is largely irrelevant to that question – it addresses a different question regarding the consequences of attending HCZ Promise Academy versus the other schools that lottery applicants attend. The data that I present are just as appropriate for assessing the claim about the elimination of the black-white achievement gap as the data reported in Fryer and Dobbie’s paper—they contrast the performance of HCZ Promise Academy students, who have been subject to the school’s “treatment,” to the citywide performance of white students in New York City and to national norms, both on the state assessments and the Iowa Test of Basic Skills, a test administered by HCZ Promise Academy to its students as part of its accountability plan. The controls for gender and income that are part of the lottery winner/loser comparison might be relevant to claims about the impact of HCZ Promise Academy on the students who attend, but I fail to see how they bear on the question of the elimination of the black-white achievement gap. I also do not see how “bubble kids” are relevant to an analysis of scale score means here—as Dan Koretz has argued, there are many ways that test preparation can result in test score inflation, only one of which involves a focus on students near a proficiency threshold.
As for the claim that I am biased because I suggest that it’s bad public policy to rely on a single assessment to make judgments about the efficacy of schools, or educational programs or policies? If this is bias, I plead guilty.
OK so to summarize: You agree with Aaron Pallas that HCZ is not closing the black-white achievement gap, as David Brooks said.
You disagree with Aaron in that you don't think we need more than one set of test scores to be sure that the academic gains the test scores in the Fryer/Dobbie study show are educationally meaningful.
So maybe now the appropriate next step for this discussion is, why do you think that one set of test scores is enough? I have to say that I am persuaded by Aaron and Dan Koretz and many other education researchers who have told me that you really need to use two sets of tests, one that is high-stakes and one that is low-stakes.
I think you misinterpret Aaron's argument on this point. He's not arguing against using test scores as a measure of academic achievement. He's arguing against solely relying on high-stakes tests.
As Dan Koretz explains it, the point of a test is to estimate the total scope of children's cognitive ability and knowledge, within a certain set of standards. But since the test cannot feasibly measure every single kind of cognitive ability and every single piece of knowledge (within that standards band), it has to measure just a sample and assume that the sample is a fair representation of the whole. When you prep children for a specific, predictable test -- as Paul Tough documented that Promise Academy teachers did -- you may end up distorting the final measure. The students will demonstrate mastery of that tiny sample of knowledge and skills, but they may not know the rest.
That's why looking at the Iowa test as a kind of check to make sure the state results are valid seems totally reasonable to me. We have to see whether the thing that Promise Academy does to a child is not just make her a better state test-taker, but also give her a stronger mastery of knowledge and skills. It seems clear that no test will be perfect at acting as a check -- as Aaron pointed out, children may not have put energy into the Iowa test or taken it seriously. And I can also imagine that the Iowa test may assess skills that we don't think it's appropriate to be assessing. But in general it does seem extremely appropriate to make efforts, at least, to verify that academic achievement is what Promise Academy added, rather than simply state-test-taking skills.
I also understand your concern that Aaron's analysis didn't replicate experimental conditions like the Fryer one did. My inclination is that it's OK that he didn't, if the question we're asking is about the achievement gap's size rather than the effect of going to Promise Academy on a child. But I am not a trained researcher so I will leave this debate alone.
The one thing I don't want to leave alone, though, is the idea that David Brooks has only 750 words and a political agenda, and that's just the way it is. I think we should do more than criticize him if he comes to the wrong conclusion. I think we should also demand high standards for what he reports as true. He reports that the Harlem Children's Zone closed the black-white achievement gap, and it didn't. He also does not give enough attention to the backdrop of social services that the Fryer study makes clear was important -- only the *combination* of that and a strong school led to these effects.
Opinion journalists should meet the same standards of truthfulness as straight news reporters like me; they are just allowed to say what they think the facts mean for public policy after they state the facts. We should be upset and hold them accountable when they fail to do so.
By the way, I want to make clear that I understand the Fryer Dobbie study shows that the Promise Academy + HCZ did do a really amazing job of *narrowing* the black-white achievement gap. I hope that the finding can be replicated with other tests and life-success measures, because that would be obviously what we all want: For the kids who weren't getting served to be better served, academically and in terms of life skills.
As a reporter covering attempts to close the black-white achievement gap, though, I think it's always important to resist the temptation to glamorize and over-state successes. And I bet that New York Times readers would be duly impressed by a finding that the gap had been substantially "narrowed."
Michael,
You're right. Brooks' column is never factually incorrect. He could have done a better job explaining the study or the HCZ, but his conclusions MAY BE accurate. The study says the large impact is from either the school or the combination of the school and all the other things.
Aaron,
Your numbers show the effects of income and racial/ ethnic achievement gaps. Because Fryer and Dobbie controlled for income, they were able to isolate the sizable black-white achievement gaps. By doing this, they were able to show that Promise Academy eliminated it. That's no small feat. It's not everything, as your analysis shows, but it's not nothing.
You also casually disregard the lottery comparison. It gives us the ability to compare two groups that are identical pre-treatment (in this case, Promise Zone). Both winners and losers came from low-income families who were motivated enough to try to get their kids in a new, different school. That is the gold standard in research, and we could only be so lucky as to have it for every study.
Elizabeth (and Aaron),
Once this debate devolves into an argument over standardized tests in general, you've taken away the ability for Fryer, Dobbie, Geoffrey Canada, and everybody else to make their case. They don't run a high school. It would be unfair (not to mention untimely) to wait to see how these effects carry over time, to see, for example, whether the kids are able to attend college in greater numbers, as Aaron suggests. Research has consistently shown fade-out for almost every valuable program. Promise Academy has shown it's very good at what it's supposed to do: raising the achievement levels of kids under its care, to standards that are set by the state.
If we want to raise standards and make a harder test, I'm all for that. In the meantime, let's celebrate the successes.
Chad,
I feel like we're talking past each other. I can admire the apparent progress that HCZ is making with its students while still raising questions about the broader policy claims that David Brooks is making based on the HCZ results.
But I disagree with your claim that there's something unfair or untimely about waiting to see if the effects of attendance persist and do not fade out before pushing a policy agenda that calls for massive investments in a program. Is there any doubt that HCZ and the Promise Academy are a costly program? If the effects do fade out, what's the long-term benefit to society of this more costly investment?
I also think that a comprehensive evaluation of HCZ Promise Academy would be better served by articulating a broader set of program goals and collecting data on progress towards those goals for lottery winners and losers. I don't fault Fryer and Dobbie for making use of existing administrative data--social scientists do this all the time--but it's a poor substitute for a real evaluation that addresses the multiple ways in which the school is intended to promote the intellectual, emotional, social and physical development of children and youth.
What we now have evidence of is great effects on one standardized test in one subject in one year for one cohort of students. And modest effects on another test in two subjects in that same year for this same cohort. I continue to think that this is a poor basis for David Brooks' claim that "no excuses" schools are the solution to the Black-white achievement gap.
It seems to me that everyone is soft-pedaling the implications of Fryer's report. He says:
"Surprisingly, there are no other statistically meaningful differences among subsets including students who entered below or above the median test score, free lunch status, and students who live within 800 meters of the boundary of Harlem Children’s Zone compared to students that do not. In other words, every subgroup gains equally from attending."
And there isn't a significant or consistent advantage to incoming students from being in HCZ. So what this really says is that, if you're only looking at two high-stakes test scores, the whole Zone is a waste of time. You only need a good middle school.
There's another aspect of this that most people tend to leave out of discussion of lottery winner-loser comparisons: Peer Effects.
(Peer effects is the well establish phenomenon of the impact of fellow students on each other.)
As a parent, if I were looking for the best school for my child, I would't want to discount peer effects. If that school over there had smarters kids with better educated parents in higher SES households, that is a something to count in its favor. I'm just looking for how to help my one kid the most.
However, from a more systemic view, we have to control for peer effects.This kids who are pushing each other -- and whose cultural capital is mutually pushing the other kids -- might otherwise be pushing other kids at neighborhood or zoned schools. The benefit that they giving each other can be concentrated in a single school, at the expense of pulling that benefit from other schools. Yes, it is better for the kids at that school, but it is worse for the kids at the other schools.
Look at Stuyvessant, the city's most competitive high school. The kids there push each other to be even better than they would at another school. But all the other kids around city would might be pushed were some of the Stuy kids to attend their own schools lose out.
I am not suggesting that we should abolish Stuyvessant, however. Rather, I am saying that if we don't consider peer effects when doing lottery winner/loser analyses, we are not actually examining what the schools are adding to the system as a whole.
Post a Comment