Big Data Analytics

What is big data?

Although there isn’t an authoritative definition of what data size constitutes big data, it is generally considered to include systems which process at least 1 terabyte of data per hour, or datasets that traditional database management tools find difficult to process & analyse. Examples of big data include search engine, traffic, loyalty card & patient data.

What big data does KCC have?

It can be divided into two categories, individual big datasets which are “big” enough in their own right and aggregated datasets which are combination of different types of data on the same customer, asset or transaction and become big data as a result.

Individual big datasets include sensor data (i.e. energy & parking meters, telecare sensors, traffic signals) and software logs (usage & performance). Big data produced by KCC customers but not captured within its systems include user-generated content (text, multimedia & location) on social networks.

Aggregated datasets include a combination of two or more of the following: transactional data (i.e. revenue & benefits), master data (i.e. customer & asset information), reference data (i.e. location), metadata, analytical data (i.e. ERP or Mosaic), multimedia, email & voice data.

What are big data analytics?

This describes the analysis of different datasets on a common type of information to identify patterns, such as health conditions, weather forecasts, or customer insight.

Why are organisations investigating this area?

The growth of big data has come about through an increase in the

1. Need for organisations to manage large sets of data through greater processing power, data

storage capacity and the cloud, the

2. Ability for users to generate content through social networks

3. Needs for organisations to track products & people on the move through sensors or RIFD

What is the business or customer need?

Big data analytics can be used to identify and uncover patterns in a variety of ways, of which these are the principal five that are relevant to local government:

1. Streamline transactions by avoiding to ask a person the same information many times 2. Profile customers to target interventions (channel shift, troubled families to health advice) 3. Analyse a location to understand its dynamics (traffic congestion/commuting patterns) 4. Investigate social media to identify local concerns (public disorder, natural disasters) 5. Aggregate sensor data to track performance (energy consumption or telecare)

KCC is analysing data on the first two areas, although research would need to be undertaken with the relevant leads in these areas to identify whether they have or need access to big data to carry out an analysis. More research would be needed to identify what teams are analysing “big data” sets, although the Traffic Management System can be considered as big data as will the NHS patient outcome reporting data that may transfer over to KCC responsibilities.

What tasks do big data analytics involve?

Big data analytics involves the following tasks: store & manage the data, process & query data and patterns of data, integrate different types of data and make statistically analyses.

What are the constraints on KCC in considering the use of big data analytics?

Information asset audits to identify high value data (and from that, big data)

Interoperability between systems to combine data

Data management, analytics modelling & business analysis skills to interpret the patterns

Information governance to assure the appropriate use of different datasets

Awareness of customer consent policies on data being analysed

Data quality to understand the reliability of data being used

What is the maturity in use of big data analytics in the public sector?

There is a strong experience of big data analytics in NHS, moderate but growing in DWP & DFT and in the

new Government Digital Service. The key factor in becoming more mature in this area is less in the choice

of big data analytics solutions to use and more the data analysis skills to be able to derive value from it.

What is the impact of big data on information architecture & governance?

As well as assuring the timeliness, quality & accuracy of data, governance of big data also requires managing its volume, velocity, and its value, particularly given the nature of big data is that it is constantly growing, being generated in real time and frequently formed of combined datasets. To be able to make use of big data analytics, there is a need to integrate the analytics solutions to existing database management & business intelligence systems, as well as end user applications.

What could be the potential next steps to investigate the potential of big data?

Understand if & in which there is a need to analyse data that is too complex or big to be currently done within KCC’s existing database management systems

Identify existing KCC systems that can manage, store, process and/or analyse big data

Understand impact on information architecture & governance

Identify if relevant skills & resources exist to make use of big data analytics tools

Identify & engage related services/organisations who have successfully used big data analytics solutions to understand lessons learned and their route map