School data: Can parents make sense of state numbers?

“I don’t know how any parent begins to make sense of this.”

That was the message Missouri Board of Education President Charles W. Shields had for Department of Elementry and Secondary Education (DESE) staff after a presentation last week on proposed levels (called cut scores) for assessing student proficiency on state tests.

“As a state board member I have been through some version of this for six years and I can barely wrap my head around it,” he added after a lengthy discussion on how to set the cut scores for last year’s MAP and End of Course exams.

So why is it so hard to understand whether or not students are failing or succeeding in English and math?

Missouri has administered three different versions of its math and English tests over the past three years with last year’s being the final version that will be used for the foreseeable future. As a result, the test scores are not comparable from year to year and DESE must create new cut scores to determine who has a below basic, basic, (both considered to be failing scores) proficient, or advanced understanding of math and English for each grade level.

These cut scores are achieved through a three-tier process the includes input from teachers, DESE staff and DESE’s Technical Advisory Committee, but it boils down to the state picking a certain score on each test that is considered the dividing line between below basic, basic, proficient and advanced performance.

In the current process, that dividing line has varied greatly between different steps of the process. For example, the percentage of students planned to score proficient or advanced on the English I EOC  moved from a whopping 70 percent (up from only 49 percent in 8th grade) in the first step to only 59 percent in the most recent version of cut scores.

“I think the cut scores are artificial,” said Shields. “The numbers are the numbers. The cut scores are contrived.”

And he is right. The students have already taken the tests. DESE’s testing company has already scored which questions the students got right and wrong. All that is left is for DESE to decide what it takes to actually pass the exam — which means they are picking the numbers that will say whether or not Missouri public schools are doing a good job or not.

In theory, the cut scores should be a big deal.

These are, after all, the only way we have of determining the quality of our schools, the only way we can compare schools and districts.

The percentages of how students in each district score as below basic, basic, proficient or advanced are supposed to play a key role in determining the district’s APR score and accreditation status.

Sadly, in reality, the only measure we have of how students are performing academically in key, foundational subjects ends up being so watered down in the final APR score that even drastic changes have little final impact.

DESE employees told the board that based on where they had currently set cut scores that some schools would see (non-comparable) drops from previous year’s results but promised that they did not need to worry because there are now even more ways that schools could get points for attendance and a “hold harmless” policy that prevents schools from being penalized for drops in test scores during the transition between tests (noting that some district were as much as 34 points away from where they should be as a result of the policy) would make the impact “really negligible.”

“Because we use a three-year average the change you may be concerned about is smoothed quite a bit,” said Assistant Commissioner Chris Neale. “Then we have the hold harmless transition that uses two-thirds of history and one third of current in order to further smooth out the transition. Anything that would be used for improvement is not affected by the scale so all we are left with is status targets and when we get the data we will decide whether freezing the status targets or recalculating them is the appropriate treatment.”

So if all of the other fluff built into the APR score calculations doesn’t keep poorly performing districts above water then DESE can just freeze or adjust the targets to make sure everything comes out fine.

So if it is hard for parents to make sense of the data, it might not really matter.