Demos: Whitehall needs to understand the limitations of big data tools

Written by Rebecca Hill on 4 April 2017 in News

Think tank says data dashboards have the potential to mislead and urges government to boost staff’s data science skills

Governments need to understand the limitations of data dashboards - Photo credit: Pexels

Government data visualisation tools are open to gaming and need to be used responsibly by people with the right training, a report from the think-tank Demos has said.

The report, published today (4 April) looks at ways to improve public services with big data, and focuses on data dashboards, which make large quantities of data, often from multiple sources, available to people.

Organisations use these tools to make everyday decisions, for instance on city planning, or to keep track of important information, such as voting information.

The UK’s Government Digital Service has built more than 800 such data dashboards for different departments, and the recent Government Transformation Strategy makes increased use of government data a priority, and pledged to appoint a cross-government chief data officer.

The Demos report said that dashboards provide a “remarkable opportunity to build more efficient and data-driven services and operations” in the public sector, but also signal a significant shift in the way government works.

Related content

John Manzoni: was a ‘misstep’ that ‘put us back a long way’
ONS launches Data Science Campus
A digital transformation for government workers

Demos said that an increased use of dashboards will create demand for new skills, the report said, while dashboards by their nature simplify information for easy digestion, which has the “potential to mislead” and presents new challenges for those interpreting the data.

There is also a risk that, “by introducing a new emphasis on metrics, indicators and measures, it can create a greater focus on operational issues rather than longer-term strategic ones,” Demos said.

To make sure governments make the most of dashboards, Demos recommended that they focus on three main areas: defining the dashboard’s purpose, understanding its limitations and training staff.

The report said that it was crucial that dashboards are “carefully designed to match real organisational needs” - without this it will be hard to judge if it is a success or to justify it to citizens who are increasingly interested in data privacy and ownership.

“With the public increasingly concerned about the privacy of their personal data, having a clear purpose for any data dashboard should also guard against 'mission creep' and make sure these large, secure data sets are only used for their original, intended use,” the report said.

“Poorly conceived dashboards are likely to be rejected by users."

When the purpose has been identified, it must be clearly communicated to developers, designers and product managers as well as the intended users, it said, adding that the users need to be engaged from the start of the design process so they buy into the dashboard.

“Poorly conceived dashboards are likely to be rejected by users,” Demos said.

It is also crucial that anyone using the information not only knows the purpose of the dashboard, but also where the data was drawn from and what underlying biases is might have - these biases need to be considered before the data is used in decision-making, Demos said.

“Users can be blinded by large numbers or have insufficient understanding of the strengths and weaknesses of the data they are using,” the report said. “Like all metrics they can be ‘gamed’, and so users must be encouraged to have a critical eye for the dashboard’s limits.”

There is also a risk of performance bias in using dashboards, due to the emphasis they tend to put on performance indicators, and can “pull your data as well as the actions of a team in a specific direction”.

A focus on metrics can lead staff to “prioritise operational issues, rather than longer-term strategic issues, and may marginalise more reflexive approaches to a problem”, it said.

‘Off-the-shelf danger’

The report also emphasised the need for more training for civil servants in data science skills.

“Just because [dashboards] are designed to be user-friendly, it is dangerous to assume users will intuitively understand how to use them,” the report said.

Staff will need to develop a whole new set of skills, including data analytics, design, social science and public policy, to make best use of the data they will soon have access to, Demos said. This includes training in how to manage dashboards.

"Bespoke dashboards can be more challenging to maintain over time."

The report also said that governments will need to think carefully about whether to use an off the shelf programme, from providers like IBM, SAP or Geckoboard, or designing something bespoke.

Although the bespoke option will allow more flexibility and a more tailored design, and are likely to be easier to update and modify, it will be down to the organisation to train people in how to use it.

“Documentation and training information can be less comprehensive or entirely absent and it can be more challenging to maintain these dashboards over time,” the report said.

But off the shelf solutions bring their own issues, with Demos saying that there is a risk of software lock-in, high costs and limitations on the appearance of the dashboard.

The advantages of commercial products often come with training courses and it is generally easier to hand over work to the next person, which is useful for organisations with high staff turnover.

However, Demos also noted that even bespoke dashboards now incorporate pre-existing software, with open source software and free visualisation tools available for low-scale dashboard design.

Share this page



Please login to post a comment or register for a free account.

Related Articles

Consultation reveals widespread opposition to proposed data-sharing laws for government login system
26 May 2023

Overwhelming majority of respondents voice disapproval but government will press on with plans to bring forward legislation

Interview: CDDO chief Lee Devlin on the ‘move from being disruptive to collaborative’
23 May 2023

In the first of a series of exclusive interviews, the head of government’s ‘Digital HQ’ talks to PublicTechnology about the Central Digital and Data Office’s work to unlock £8bn...

HMRC finds strong support for online Child Benefit claims – but ‘digital by default’ would cause problems for one in five users
17 May 2023

Department publishes findings of study conducted ahead of planned digitisation initiative

Digital Leaders’ Download: Ex-HMPPS leader Farrar on how technology was crucial in helping prisons’ pandemic response
31 May 2023

In the first of a series of interviews with government’s biggest figures, PublicTechnology and CDDO caught up with  Jo Farrar to discuss exploring virtual reality and AI, and why it’...

Related Sponsored Articles

Proactive defence: A new take on cyber security
16 May 2023

The traditional reactive approach to cybersecurity, which involves responding to attacks after they have occurred, is no longer sufficient. Murielle Gonzalez reports on a webinar looking at...