School leaders use value-added systems to fine-tune instruction

By Del Stover

01/18/05 -- When school officials in DuBois, Pa., took a closer look at the year-by-year academic progress of individual students, they made some important discoveries about their instructional program.

One of the more noteworthy findings was that higher-achieving students at some schools weren’t making the kind of progress that they should.

“Even though they were scoring okay on state assessments, they weren’t making the growth [expected] in a given year,” says Karen Foster, curriculum director for the 4,500-student school system. “Especially in math, we were having them repeat a lot of the curriculum.”

Every school district attempts to measure the academic progress of their students and schools. But the DuBois schools are participating in a pilot project of the Pennsylvania education department that provides a “value-added” assessment of student test scores.

With this approach, the progress of individual students is measured as they move from grade to grade, and after making comparisons with the progress of students elsewhere across the state, this assessment model more accurately measures how well students -- and schools -- are performing.

“It brings a couple of pieces to the accountability system that don’t exist with other calculations,” says Mitchell Chester, assistant superintendent for policy and accountability at the Ohio education department, which is piloting its own value-added assessment program in 78 school districts across the state. “Many will view it as a fairer assessment of how schools and districts are doing.”

Both Pennsylvania and Ohio plan to expand their value-added initiatives statewide.

The demand for a fairer assessment tool is a major reason why the value-added assessment model is sparking interest these days from educators and policymakers across the nation. Last year, more than a dozen state schools chiefs asked the U.S. Education Department to allow states to use a “growth model” in determining a school’s academic progress.

Some educators are highly critical of the assessments currently used to determine annual yearly progress (AYP) under the No Child Left Behind Act.

Analyzing a school’s third-grade test scores from one year to the next is a comparison of two entirely different groups of students, and thus offers few clues about the actual progress of students.

The AYP formulas now used also make no allowance for schools that serve a high percentage of disadvantaged students who enroll in school far behind their more affluent peers. And schools with high student mobility are judged on the performance of students whom those schools have had no opportunity to influence.

In contrast, value-added models judge the progress of the same cohort of students over time. Data for a school with low overall test scores might, in fact, reveal that although students started off far behind the state average academically, the school is making great strides in closing the achievement gap.

Not only does this information provide a more accurate picture of a school’s instructional program, a detailed breakdown of the data can prove a valuable diagnostic tool for schools and districts, says Suzanne Amos, spokesperson for Battelle for Kids, a nonprofit education reform group overseeing Ohio’s pilot program in value-added assessments.

“If you look at the progress individual students and groups of students are making year to year, you might find an affluent district that scores very high on achievement but has students who are not making the progress they need to be making,” she says. “It’s an additional piece of information to make better data-driven decisions.”

Such information led to a number of changes to the instructional program at DuBois. Early findings indicated that some schools needed to offer a more rigorous program for top-performing students, and officials responded with more teacher training, more enrichment programs, and, at the high school, a push by teachers to encourage participation in Advanced Placement and other challenging courses.

At other schools, the data showed a lack of progress among some of the lowest-performing students. After teachers at those schools modified their instructional practices, there was a noticeable gain in student test score averages. One teacher raised student performance in her class by 25 percent.

“Value-added gave our teachers a different side to look at in regard to their student achievement,” Foster says. “It added more depth to the analysis of our test data.”

That’s also proven true in Chattanooga, Tenn., where Kirk Kelly, director of accountability and testing, says the Hamilton County school district uses a value-added assessment system to identify where resources, training, and interventions are needed.

The data also is used in a pay incentive program designed to attract highly qualified teachers to the district’s lowest-performing schools, where teachers who show the highest test-score gains can earn an extra $5,000 annually.

Tennessee was the first state to adopt a value-added assessment system in the early 1990s, and only now are other states beginning to jump on the bandwagon. For now, the two states with pilot projects -- Pennsylvania and Ohio -- intend to use the assessment system as a diagnostic tool, although Pennsylvania officials are looking at the potential to use value-added data as an official part of its accountability system.

That would require the approval of the U.S. Department of Education, which so far has rejected proposals to incorporate value-added assessments into AYP calculations.

Yet, NCLB also has given the value-added model a significant boost because, for the first time, states are required to test students on an annual basis, providing educators with the year-to-year test data needed for a longitudinal study of individual student progress.

For local school officials hoping to take advantage of this data, however, there are some pitfalls to avoid, says William L. Sanders, architect of Tennessee’s value-added assessment system and a pioneer in the field.

“All value-added models are not created equal,” he says. “It’s important to not fall into the trap of using some of these overly simplistic models, where you get misinformation instead of reliable information. You don’t want to make decisions on information that’s not reliable.”

Some educators have raised concerns about the reliability of these models, questioning the statistical methodologies used and the ability of those analyzing the data to account for changes in test questions, missing test data from year to year, student mobility, and the background of students.

And some educators suggest the complexity of the data analysis is beyond all but the largest school systems. Smaller districts that want to take advantage of the system can turn to a state-sponsored program or a private firm, such as SAS, a business software firm where Sanders serves as manager of an assessment service for more than 400 school districts nationwide.

Robert Lissitz, director of the Maryland Assessment Research Center for Education Success at the University of Maryland, says small districts attempting to track student growth on their own should be wary of drawing any hard conclusions from the data.

This analysis of data can be useful, but it does not take into account a host of variable factors and should not be used as the sole means to determine teacher quality or the success of an instructional program.

“It’s complicated,” he says. “We don’t have a model that allows you to tease out some variables about teachers and policies.” The data is good enough to suggest a closer look at a particular school or program, he says. But to use the data to draw firm conclusions is “not realistic.”