An Unstandard Deviation

On February 20th 2009 Beverly Hall, then in her ninth year of heading Atlanta’s public schools, was named National Superintendent of the Year by the American Association of School Administrators (AASA). The head of AASA praised the “significant gains in student achievement” she had overseen—and indeed Atlanta’s schoolchildren made sizeable gains on the standardised tests used to determine yearly progress under the federal No Child Left Behind Act. At Parks Middle School, for instance, the share of 13-year-olds who passed the test’s maths section rose from 24% to 86%, and the share of those who “exceeded expectations” rose from 1% to 46%—both in a single year. The state of Georgia alleges that those remarkable leaps rested on neither pedagogy nor determined study, but something far more invidious: cheating.

So begins a story in the April 6th 2013 edition of The Economist that highlights the power of basic statistics to detect signals from noise — in this case, a signal that turned indicated a too-good-to-be-true situation. The cheating was not by students but by the school administration and teachers who either gave students the answers to questions or erased incorrect answers after the test and filled in the correct ones. The investigative team

flagged classrooms with an average number of wrong-to-right erasures more than three standard deviations above the state average. The chance of that occurring randomly is one in 370. More than half of Atlanta’s elementary and middle schools had such classrooms, and many had erasures more than 20 to 50 standard deviations above the norm.

The cheating did not help the students at all, because “..schools that cheated their way to false improvements lost federal funds which could have been used to make actual improvements. Because of their apparently high test scores, struggling pupils were denied the help they needed and deserved.”

Also, once again, we see an example where a basic understanding and application of statistics is a powerful tool for managers and citizens alike.

Refresher question for your Lean Six Sigma Black Belts out there: is the 1 in 370 ratio the article cites correct or incorrect?