Cabinet Office taps data analytics firm for ‘spring clean’ of millions of files

Written by Sam Trendall on 21 February 2023 in News

Department extends existing supplier engagement in £500k deal

Credit: tookapic/Pixabay

The Cabinet Office has signed a £500,000 deal with to expand the department’s use of automated data analytics to support “spring clean” exercises in which millions of files will be sifted – and, where applicable, deleted.

Freshly released commercial documents indicate that, two years ago, the department “decommissioned a legacy IT system called Apollo… [which] contained significant volumes of unstructured and semi-structured data” – amounting to about 4TB, across 11 million individual files held in 170 differing formats.

Initial analysis suggested “there were significant volumes of ROT – redundant, outdated, trivial (ROT) – [information] as well as historically valuable information that must be retained by the Cabinet Office for long-term preservation”. 

On 1 February 2021, the department signed a contract with Belfast-based data analytics firm Automated Intelligence. In a £124,000-a-year deal that ran until 31 January 2023, the firm’s AI.Datalift technology has been used by the Cabinet Office as the foundation for developing an “automated decision-making tool… [that] was piloted on the legacy Apollo data set”.  

“The tool has proven to deliver accurate and reliable disposal decisions, aiding digital archivists to greatly reduce the volume of ROT held by the Cabinet Office for this data set,” according to newly published procurement documents.

Related content

Senior executives on the department’s People and Operations Committee last year approved an expansion of the automated analysis tool for use on other data sets held by the government's central agency.

The Cabinet Office has thus awarded a new two-year deal to Automated Intelligence, running directly on from the conclusion of the previous deal on 31 January. This contract will be worth £236,500 a year plus VAT – almost double the value of the prior engagement.

The contract outlines that the firm is expected to provide “data analytics tool that is programmable so that data can be classified as either ROT or as valuable… [and] that is able to action a deletion on data so designated”.

The system developed using the company’s technology will now be used to support annual “spring clean” projects in which millions more files will be analysed.

These exercises – led by the Cabinet Office’s Digital Knowledge and Information Management (DKIM) team – are designed to sort through digital documents and categorise them for either deletion or retention.

This process has created an accumulation of “legacy information awaiting a disposal decision”.

“DKIM currently holds 4.9million digital files in its ‘holding pens’ which is made up of information collated from across Cabinet Office through the annual DKIM-led spring clean process, an annual muster of records identified for disposal and retention according to Cabinet Office policy,” the text of the new contract says.

Automation technology will be used to further analyse files that are set to be retained, in order to identify and remove any remaining redundant, outdated, and trivial data.

Having completed sifting of this legacy data, the system will be used to “analyse information collected through the DKIM spring clean in future years, currently estimated to be 350,000 documents per annum”.

The deal with Automated Intelligence will see the company work with the Cabinet Office to “explore the use of the tool to support business units to identify information in scope for future spring clean activity – [which] is currently a resource intensive, manual process”, the contract says.


About the author

Sam Trendall is editor of PublicTechnology. He can be reached on

Share this page




Please login to post a comment or register for a free account.

Related Articles

Research firm brought in to improve assessment of major tech projects
17 March 2023

Government’s Evaluation Task Force has signed a deal with Ipsos to address ‘significant lack of good quality evaluation’

Government to create departmental savings tracker as Budget signals spending squeeze
16 March 2023

Treasury will help track efficiencies delivered through technology and cost reductions 

NCSC probes TikTok amid reports of imminent ban of government devices
16 March 2023

Security minister confirms intelligence agency is investigating the video app

Worsening skills shortages threaten government digital transformation, NAO finds
13 March 2023

Auditors praise the ‘fresh approach’ of CDDO but warn that unit’s work across government could be compromised by access to expertise

Related Sponsored Articles

Digital transformation – a guide for local government
6 March 2023

Digital transformation will play a key role in the future of local government. David Bemrose, Head of Account Strategy for Local Government at Crown Commercial Service (CCS), introduces a new...