McKinsey Global Institute ranks it as one of the largest sectors of growth in the coming years, describing "a tremendous wave of innovation productivity and growth...all driven by Big Data". The Economist World in 2012 proclaims that "2012 will be the year in which the Big Data trend gets noticed". Big Data is already here, and one of the key challenges now is determining how it can be brought to those to whom it could bring real positive benefit.
Big Data - the ability to process, analyse, and understand vast amounts of data - has long been the purview of insurance firms, cashed-up hedge funds, and tech juggernauts such as Google, Yahoo and Microsoft. These companies were, for a time, the only ones who had access to such large amounts of data and knew that they could benefit greatly from analysing it.
But this is no longer true. Most companies now realise they have huge quantities of data and records, sometimes only in paper form, that do little more than take up room in archives. In a world of data-driven decisions, this is a lot of value that is just left on the shelf.
This is a tragedy that is especially true in the charitable sector. Social enterprises and NGOs are generating piles of medical data, quality of life surveys, and micro-lending transaction data faster than they can properly make use of it. But with a looming 100,000 person talent gap in the us alone*, there will be 100,000 big-data-related roles that we don't have the expertise to fill, and charities and NGOs risk being left out in the cold. The ability to process big data could make all the difference for charities, bringing them into the new realm of data, increasing efficiency and deploying resources where they can have the greatest impact.
However this doesn't have to be the case. Data scientist Jeff Hammerbacker (CEO of Cloudera and one of the first 100 facebook employees) famously said, upon leaving the social media giant: "The best minds of my generation are thinking about how to make people click ads. That sucks." This is a widely-shared sentiment in the Big Data community, that talent is increasingly drawn to where the most money is (online advertising) and away from where it could have the most social impact.
There is a rapidly growing Big Data community and many of its members want to donate their resources to aid charities and NGOs in dealing with their data. However, the gap between the charities with the data and the analysts who can unlock its potential remains dishearteningly wide. With a lack of widespread knowledge of Big Data and few existing links between the two communities, there have been few avenues for productive collaboration. This partnership between analysts and charities is where the key investment needs to take place if the social sector is to ride the wave of big data, bringing together the analysts who want to assist the third sector with third-sector parties that could benefit immensely. While there is still a lot of work to be done, some groups are already taking on this challenge, and two of the most promising are Data without Borders and Kaggle.
Data Without Borders was founded last year by New York Times Data Scientist Jake Porway in response to exactly this problem. He saw the chasm between charities and the Big Data community and started working to form real links between the two groups. Data without Borders has already organised multiple events that brought charities together with Big Data scientists who could help them unlock the potential of their data. Initial projects were run with charities such as UN Global Pulse and The Microfinance Information Exchange Market. If Data Without Borders can reach a critical mass where many charities are aware and willing to partner their data, they could easily become the global force for charitable data analysis.
Kaggle is a less obvious project to bring data expertise to social enterprise. Kaggle does not focus on charities or NGOs. Rather, it runs competitions where 1000s of data scientists compete to provide the best solutions to problems with data. Kaggle has already achieved fame for helping NASA to map Dark Matter in the universe. Within only weeks, competitors unfamiliar with astronomy were able to significantly outperforming the most cutting edge models for this cosmology problem. The platform they've created for competitions has the potential to instantly overcome the problem of bringing the two communities together.
While they have yet to run a competition on behalf of an NGO or charity, the platform shows great promise for rallying data scientists behind the larger data problems that these organisations face. While other competitions have traditionally offered cash prizes for the winner, charities and NGOs can appeal to the unsatisfied desire of data scientists to use their skills for the greater good. And it is there. Once charities and NGOs gain a clearer understanding of what they want to extract from their data, this will provide an amazing means to bridge the gap between data-laden charities and analysts with an appetite for charity.
This is the beginning of a new era in the use of information and it is up to us to make sure that the third sector isn't left behind.