It's no secret that we're creating more and more information both in our personal and business lives. Indeed, more than 7 terabytes of tweets are created on Twitter every day, and this figure stands at 10 terabytes for Facebook. The real question is how can we keep on top of this information and make it work for us? And, is it really worth the effort? It's certainly a challenge, but most companies today will recognise that there is potentially reams of hidden value in the data their employees create, use and store every day, but when it comes to actually unlocking this value, it's often a different matter altogether.
So what is big data? It's an approach whereby businesses are no longer relying on a small number of well-curated sources of information when working out how their business is performing. Those sources are still used, but they're also taking into account newer and more ad-hoc sources and attempting to fuse them together. And although it's called Big Data, we're not only talking about volume - although data continues to grow at a bewildering rate - it's also about the variety of sources being tapped and the speed at which the data comes at you. The desired result is a more rounded picture of how the business is performing, right now.
For example, if you want to know what consumers are saying about your product, then Twitter, Facebook or other social networks might be the best place. If you're trying to analyse employee morale, then email, internal instant messaging might be the best source. If you're trying to uncover fraud, then analysing trends in claims documents and contracts might be the best source. The point is, all the answers aren't in one place, and they could be anywhere. So why wasn't this done before? Quite simply, the software wasn't available and computers weren't powerful enough. Well, it is now and they are.
But Big Data offers so many more possibilities beyond mere improvements in targeted marketing. For example, the World Health Organisation now monitors Twitter and other social networks to extract intelligence and identify trends related to global health concerns. This information can then be used to inform policies, and ensure the public is fully briefed on the situation in as timely a manner as possible. Also, wouldn't it be great if handwritten notes by doctors and nurses could automatically be incorporated into your online medical record? It would certainly help the next doctor we see with when it comes to diagnosing and prescribing medicines, and it would also provide peace of mind to patients that they're getting the best service.
Thinking about other organisations that could make better use of Big Data, what if a bank could work out that one of its customers is having money worries and actually work with the individual to create a new payment plan in line with their situation? Maybe the customer in question has just phoned up a different department to take out a loan or apply for an extended overdraft limit - by applying intelligent analysis to this situation, the bank could then notify the account holder and extend the payment period by a few days or a week. The result would be a happy and loyal, rather than a stressed out and anxious, customer.
But in reality, most banks aren't in a situation where they could do this even if they wanted to. Customers no longer communicate with businesses via letter or fax, sent into corporate headquarters. In addition to phones, they are now turning to Twitter, Web forms, email and texting to interact - making it much more difficult for banks to have a complete view of each account holder because of the numerous interaction points, which means they're not in a position to accurately 'listen' and act on the information customers provide.
The same goes for most service providers, whether they're a mobile phone provider, utility supplier or an insurance company. What's more, by only hearing part of what your customers are saying, businesses are not only potentially turning off their customers, but they could also be missing opportunities to upsell, or worse, letting fraud go unnoticed.
Embracing the big data ethos means listening to those newer and more varied sources of information, as well as traditional means of communication and tying it together with data from transactional systems.
By tapping into those sources and making it available for analysis, organizations can unlock the real value in big data. Big data can be messy, but businesses can learn a lot from messy data, because insight can come from the most unexpected places.