The Government and public sector have a head start on many other organisations as they already have a rich source of data. The trouble is that all that potentially valuable data is not shared, but locked away in departmental silos. This antiquated approach is blocking the benefits that would arise from integrating data across departments to give a joined-up view of people, operations and services.
Imagine every time a citizen had contact with Government that one single view of that citizen was available, rather than the citizen having to give the same details every time. It would be a win win - citizens would benefit from a more streamlined, tailored service, and not be bothered by unnecessary requests for information already released, while the Government could uncover new and more accurate insights from a larger pool of data.
Big data analytics can be used to understand what data is important, and transform it into insight for better decision-making, increased efficiency, faster innovation and improved services.
However, a reasonable balance is needed between civil liberties and data protection when considering how to make data more accessible. Whilst endless possibilities may exist from more data sharing, opportunities will be justifiably limited by measures to safeguard sensitive information that doesn't need to be shared in a particular situation. Indeed, the views of individual citizens should be taken into account - in particular, whether they are comfortable for all their data to be shared.
Exceptions to whatever is the agreed status quo would have to be made where data needs to be accessed to assist with certain activity such as law enforcement, counter-terrorism or issues affecting national security. Subject to that, the preferences of the individual should be honoured. There could even be a system established where citizens are actively asked every, say, five years what information they are prepared to share.
Public sector organisations cannot claim a general "public interest" defence to releasing data in the same way the media can, yet there may be scope for civil servants to be given more flexibility in terms of what they can and cannot share - e.g. it can be done where it protects the person's identity unless there are clear or serious legal reasons not to, or the individual has consented to share that data.
It need not be a simple choice between sharing all data and sharing no data at all. Gradual sharing of data, where relevant, could be achieved by a very small group of people first assessing the usefulness of data to different departments, so as to assess the merits of joining up data sets versus risks to civil liberties. Where that balance falls might change over time but this approach can be refined accordingly and ensure good stewardship of data. This analysis could also reveal areas where it's best for data not to be shared - and, alternatively, areas where sensitive information may need to be shared where public safety, for example, is at risk.
Even if just a proportion of the population provided broad access to their personal information for government use, then it would create a valuable resource to create models and derive important insights. It's not vital for everyone or even the majority to agree to share their data. Sharing clinical trial data is highly valuable for making discoveries that can lead to important breakthroughs in treating diseases, yet only a small proportion of the population will have taken part in these trials.
The advantage for government over commercial enterprises is it can potentially access sufficient data to build up a much better overall view of the population. If you compare supermarket loyalty schemes, they are limited to analysis of their customer base - yet they have still been very successful in enabling them to respond to the needs of customers and foster loyalty.
It would be wrong to simply look at ways of sharing the data itself. Some departments might carry out all their analytics on a single platform for a single source of the truth to serve policy decisions, or operational and internal reporting. It might then be possible to share insights from the analytics with other departments, rather than simply opening up access to data. There could also be sharing of models - there would again be no need to share data, as it could be a case of simply running the same model on another department's data.
A further option is to create models from de-identified designated data, so the individuals involved cannot be identified. When models are deployed against the complete data set and cases emerge that require investigation, the data concerned is then unlocked and made available. This is no different to the previous world where investigators could pick up individual files at random for investigation.
There are competing interests and lots of options as outlined above. What's clear is that data sharing is not black and white - it can be achieved to varying degrees and in many different ways. Which means the government has many different pathways open to it for progress in this area - any data sharing is a big improvement on very little or none at all.