States vary greatly in their requirements for what students need to know to score at the proficient level on state tests, according to The Proficiency Illusion, a new report from the Thomas B. Fordham Institute and the Northwest Evaluation Association (NWEA). The report finds that, because of this variance, the results on state tests are creating a false impression of success, especially in reading and in the early grades.
“NCLB rests on quicksand. We’re supposed to think that it is providing greater transparency about the performance of students, teachers, and schools,” said Chester E. Finn Jr., president of the Thomas B. Fordham Institute. “In fact, the very opposite is happening: proficiency standards in math and reading vary erratically, almost randomly, from state to state, grade to grade, year to year. Parents cannot be sure that they are getting accurate feedback on how their children are really doing in school-or how their kids’ school, school system or state is really doing.”
In performing the analysis, NWEA experts used a mapping exercise to project state “cut scores,” or the level needed to pass the test for the purposes of the No Child Left Behind Act (NCLB), onto the scale used by the Measures of Academic Progress (MAP), a test used by twenty-six states for diagnostic and accountability purposes. Specifically, the NWEA researchers used the point on a state test at which 70 percent of the study group performed above the state proficiency cut score to the point on the MAP scale that was met or exceeded by 70 percent of the study group.
The decision to use the MAP test over the National Assessment of Educational Progress (NAEP) was made because NAEP tests are given only at three grade levels-fourth, eighth, and twelfth-and because of questions about student motivations for NAEP, which does not report individual- or school-level results. MAP, on the other hand, is given in grades three through eight, is reported to parents, and is aligned to individual states’ curriculum standards.
In examining grade three reading cut scores, NWEA researchers have found that states ranged from the seventh percentile (Colorado) to the sixty-first percentile (California) on the NWEA scale. The report notes that the difference in expectations are “striking” and adds that the “difficulty in the reading passages … range[s] from something that could be found in a second grade reader to a passage from Tolstoy.”
At the eighth-grade level, the score at which a student would be considered proficient on a state reading test compared to the equivalent score on MAP ranges from the fourteenth percentile (Colorado) to the seventy-first percentile (South Carolina). The report also finds that twenty-three states possess an average score required for proficiency that would fall below the fiftieth percentile on MAP.
NWEA experts also examined math proficiency cut scores in grades three and eight. They find that the range of proficiency cut scores in third grade math went from the sixth percentile (Colorado and Michigan) to the seventy-first percentile (South Carolina). At the eighth-grade level, proficiency cut scores in math ranged from the twentieth percentile (Illinois) to the seventy-fifth percentile (South Carolina).
Using the average rank of state cut scores across all grades, the report indicates that Maine, California, and South Carolina generally have the most challenging tests in reading, whereas Colorado, Wisconsin, and Michigan had the least challenging. In math, California, Massachusetts, and South Carolina have the highest standards while Colorado, Illinois, and Michigan had the lowest.
The report also finds that standards for mathematics are generally more difficult to meet than those for reading. For example, in Massachusetts the math cut score is set to the sixty-seventh percentile for eighth-grade students, but the reading cut score is only set at the thirty-first percentile. Consequently, 74 percent of students achieved proficiency in reading, but only 40 percent achieved proficiency in math. The report notes that such disparities can have huge implications for policymakers.
“School systems and policymakers may infer from the resulting differences in performance that students in a given state have some deficiency in mathematics requiring special intervention,” the report reads. “They may act on these kinds of inferences, allocating resources to address seeming gaps in math achievement that may not exist. As a consequence, resources might not be allocated to address problems with reading programs that remain hidden beneath this veneer of seemingly superior performance.”
NWEA experts also find that disparities exist in the tests given in the earlier grades versus those given in the later grades, with reading and math tests in the upper grades consistently more difficult to pass than those in the earlier grades. Using Minnesota as an example, the report finds that a student who performed at the thirty-fifth percentile on MAP would have been proficient in grades three, four, and five, but not proficient in grades six, seven, and eight.
In discussing what the report means for policymakers, Finn and Michael J. Petrilli, vice president for national programs and policy at the Thomas B. Fordham Foundation, argue in the foreword to the report that the “testing infrastructure on which so many school reform efforts rest, and in which so much confidence has been vested, is unreliable-at best.” They add that Congress erred “big time” when NCLB allowed each state to set its own standards and devise and score its own tests.
In offering an alternative to the current environment, Finn and Petrilli say that it’s “not crazy” to have some form of national standards. They suggest that the federal government does not have to set national standards, but that it should at least be responsible for ensuring that they get set. They note that the House Education and Labor Committee draft to reauthorize NCLB includes “financial incentives for states that adopt ‘world-class’ standards that imply readiness for work or college,” but also undermines this objective by “slavishly clinging to the ‘100 percent proficient by 2014′ mandate.”
“If Congress wants states like Michigan to aim higher,” they write, “the best thing it can do is to remove this provision from the law. With this perverse incentive out of the way, Michigan just might summon the intestinal fortitude to aim higher-and shoot straighter.”
The complete report is available at http://edexcellence.net/institute/publication/publication.cfm?id=376.