A few additional thoughts on the post below about margins of error, etc., based on the comments in this and earlier posts. (An aside: My take on anonymous commenting is pretty much akin to my take on anonymous blogging: Why? It's a free country, nobody's going to yank you out of your house in the middle of the night because of what you write about No Child Left Behind. Identify yourself, we'll all be better for it. Particularly, I should add, if you're claiming some kind of expertise. If you're really a statistics professor, say so; I'm not giving handles like "StatsProf" any more credence than "Anonymous" or "SomeGuyYellingOnTheStreetCorner.")
First, some people have raised the issue of measurement error vis a vis the tests themselves. And it's true that (A) tests only cover a fraction of what we want to students to know, and (B) tests are an imperfect measure of that fraction. But the place to make allowances for that should be the process of setting cut scores. States don't require students to get 100 percent of the answers right to be deemed "proficient." (In fact, all states have a separate, higher "advanced" level which is also less than 100 percent.) Once states determine what less-than-100 score is proficient, it makes sense to build in some cushion to account for imperfections in the instrument, random variation, etc. This is particularly true if the test is being used for individual high-stakes purposes, i.e. graduation and grade retention policies. But having made those allowances, don't layer on new allowances with confidence intervals and tell the public that students are meeting NCLB performance thresholds when in fact they're not.
Second, there's certainly no unanimity in academia on the issue of confidence intervals. Some very qualified, well-trained people see it one way, some see it another.
Third, what's consistently lost in all of these discussions is the need to balance the risks of Type I and Type II errors. When implementing NCLB over the last six years, states have been almost exclusively interested in making sure that good schools aren't mistakenly identified as "in need of improvement" or "failing" under NCLB. And that's a legitimate thing to worry about. But they've all but ignored the risk of under-performing schools being mistakenly not identified as needing improvement. There's no way to completely eliminate both risks; states have to find a reasonable middle ground between them. Instead, many states have been obsessed with finding new ways to make sure there is absolutely no possible way for a good school to be falsely identified as underperforming, and in doing so have knowingly let schools and districts that are obviously not serving students well slide by for years and years. Of course, focusing only on minimizing the first kind of risk makes sense if you view NCLB as fundamentally illegitimate and malign, but if that's the case just say so and stop pretending your concerns are limited to proper use of statistics.
Subscribe to:
Post Comments (Atom)
2 comments:
..."knowingly let schools and districts that are obviously not serving students well slide by for years and years. Of course, focusing only on minimizing the first kind of risk makes sense if you view NCLB as fundamentally illegitimate and malign, but if that's the case just say so ..."
Here we agree. NCLB-type
accountability is fundamentally illegitimate and malign, for three reasons at least.
1. We have a relatively easy way around statistical games that would probably, in itself, produce far more accountability. Remove sanctions in return to having states follow professionally accepted standards. Get rid of NCLB's over-arching data-driven accountablity and you may see a golden age of focused data-driven decision-making. But the two are largely incompatable.
2. Data and accountability and statistical procedures differ depending upon your focus. One margin of error is acceptable for creating a Consumer Reports - type ranking. The same margin of error would be ridiculously beyond the pale when the careers of individual teachers are on the line. But due to the nonstop conflict between people who should be united, those distinctions can be lost when the blame game gets in high gear.
3. Those obviously bad districts and teachers will continue to "slide" along as long as we have the incompetent accountability of NCLB. Their paths will get rockier, and some people can get happy by attacking others, but no real progress will occur until we stop fighting each other, build capacity, and design rational AND FOCUSED accountability systems.
Had we passed NCLB I, with all its flaws, but without AYP, the law would have done more good - if only by accident. If you really believe in accountability, hold yourself and your political allies accountability for devising a rational system, not one that tarnishes the whole concept of accountability.
Yes the educational status quo has malign effects on poor kids. But defending the indefensible and attacking people who have dedicated their lives to children is not a legitimate reform.
john thompson
There is no single rationale behind the cutpoints. Hence, adjustments for error make no statistical sense. However, these adjustments make as much sense politically as any other reason for cutpoints -- they are just another way of lowering the threshold. In the end, schools just get told to meet some standard, whatever the reasons. As many of the other rationales are as hypocritical and pseudo-scientific as the statistical adjustments, why save indignation for the latter?
Post a Comment