Computer modelling process to be examined after it emerged that Ofqual turned down help from experts that refused to sign NDA

Credit: Pixabay

The UK’s stats watchdog has said it will conduct a review of the statistical models used by qualifications bodies to award A-Level and other exam results, amid an ongoing row.

The Office for Statistics Regulation said it would also be looking into the processes by which Ofqual and other qualifications regulators decided on the models used to hand out grades to students.

The news comes after a major U-turn, in which the government said it would ditch the controversial algorithm used to calculate this summer’s A-level and GCSE grades across England.

But in a letter to the Royal Statistical Society, which has been calling for a review, the OSR said there was “still value” in the exercise despite the change in approach.

“OSR therefore plans to undertake a review focused on the process of developing the statistical models. Our review will consider the extent to which the organisations developing the models complied with the principles set out in the Code of Practice for Statistics,” Ed Humpherson, its director general for regulation, said.

The probe will aim to “highlight learning from the challenges faced through these unprecedented circumstances”, and will not address the implications of the algorithm’s use on individual results, he said. Nor will it take a view on the best way to award grades in the absence of exams.

Related content

• Sturgeon apologises after outcry over exam results downgraded by ‘deprivation algorithm’
• No plans for ‘right to challenge’ laws for algorithm decisions
• Home Office halts use of visa algorithm after legal challenge to ‘racist’ system

“There are many areas of the approach to awarding exam grades this year that may warrant review and it is likely that other organisations will commission or carry out reviews. We are conscious that too many reviews could be unhelpful and will seek to minimise overlap between our review and others,” Humpherson added.

He said the OSR would therefore “try to minimise the burden of our review on organisations involved in awarding exam grades” and would contribute its findings to other relevant reviews where appropriate.

Responding to the letter, Sharon Witherspoon, RSS vice president for education and statistical literacy, said: “The lack of transparency around the process has not only caused significant distress for thousands of students, it has threatened to undermine public trust in statistics and their use. It is therefore right that the Office for Statistics Regulation looks into these issues to ensure this does not happen again.”

Non-disclosure agreement
News of the review comes shortly after it emerged that Ofqual had earlier turned down an offer from the RSS to provide independent expert help with the algorithm because the statisticians in question refused to sign non-disclosure agreements.

The royal society wrote to the exam regulator in April with a warning about some of the challenges involved in estimating student grade, and calling for an advisory panel to be created. That letter, written by Witherspoon, put forward independent statisticians to sit on such a panel – but Ofqual said, to do so, they would need to first sign an NDA.

Witherspoon said Ofqual faced two major issues in deciding how to calculate results: data collection and quality; and the subsequent statistical modelling of anticipated grades. 

“These are objectively difficult decisions by any standards,” she wrote. “The Royal Statistical Society is ready to help if that would be useful. We believe that having an engaged expert advisory group would be of use as you make decisions, and would be preferable to a purely internal process.”

She put forward two RSS fellows: Guy Nason, a stats professor at Imperial College who already sits on a Department for Education advisory group on higher-education figures; and Paula Williamson, a professor of medical statistics at the University of Liverpool.

But in a letter to the Office for Statistics Regulation last week, Witherspoon and RSS president Deborah Ashby said Ofqual had insisted the two sign an NDA “that gave us real concern”.

The letter said advisers should not give a running commentary nor divulge confidential information, and that Witherspoon had said as much to Ofqual. “But the proposed confidentiality agreement would, on our reading, have precluded these fellows (who were suggested precisely because of their relevant statistical expertise, and lack of ties to qualification regulators or exam-awarding bodies) from commenting in any way on the final choice of the model for some years after this year’s results were released.”

Witherspoon said that she had written to Ofqual outlining her concerns with the NDA “and restated our willingness to help if a more suitable agreement could be reached”.

But the letter, dated 14 August, added: “We did not get an official response to those questions, and our offer to help was not taken up.”

A spokesperson for the qualifications regulator said it was “routine” for Ofqual to use NDAs when handling “sensitive information which it considers requires protection for reasons in the public interest”.

They said: “We published the precise details of the standardisation model, as we said we would do, in addition to a number of technical reports. Throughout the process we have had an expert advisory group in place, first meeting with them in early April. The group includes independent members drawn from the statistical and assessment communities. The advisory group has provided advice, guidance, insight and expertise as we developed the detail of our standardisation approach.”