In life, tough decisions have to be made; someone is chosen to get a job while others are not, you either qualify for a mortgage or you don't. Mortgage decisions used to be made using human intuition - customers had a personal relationship with their bank manager and he decided whether they were a good risk or not. But this system was expensive to run and bank managers' decisions were biased, so instead banks started relying entirely on data so that they could make objective predictions. Until recently this has primarily been data from application forms; a mortgage application asks questions about a prospective borrower's income, job, and lifestyle and everyone understands that this information is used to make loan decisions.
Now, not just banks, but employers, advertisers and retailers know that there is data out there about us that will help them to make even more accurate decisions. Every time we use a social network, search engine, web browser, cellphone, or credit card, we leave a trail of data behind us that can be collected, stored, shared, and used to make predictions with. My research has shown that even apparently innocuous Facebook Likes can be used to accurately predict personality, IQ, political views, religious views, and sexuality. Imagine the predictions that could be made from the whole of someone's data trail.
There are reasons this is a good thing. Banks could give loans to young people without a credit history but whose social media data predicts that they are trustworthy. Car insurance companies could give lower premiums to drivers whose shopping habits suggest are less impulsive and prone to anger. Recruiters could find employees whose interests and values match the jobs that are available.
The risks are that companies are starting to do this already but consumers are unaware of how their data is being used. It might be that you are denied a loan but you never find out that it was because of your Twitter profile. Or perhaps the company doesn't even send you a loan advertisement in the first place because they've already predicted that you won't be accepted, so you never even realise the opportunity you missed out on. This is dangerous because how can each of us react to the predictions made about us if we do not even realise they are being made?
The biggest problem right now is that consumers do not understand how or when their data is being used and this leads to fear of the unknown and a feeling of powerlessness. From talking to many tech employees, my experience is that they are well-intentioned and are only trying to improve our online experiences, but unfortunately companies are not transparent with how they use data so consumers often find it 'creepy.'
I am not a refusenik. Advancements in digital services have enabled incredible services to exist that wouldn't otherwise. Every so often I remind myself how absurd it is that search engines give us access to the entirety of human knowledge in milliseconds. I believe that current digital services and systems can be improved to make people comfortable with how their data is used.
Companies should tell us when they make predictions and explain them in an understandable way. For example, when you see an advertisement you should be able to find out why you've been targeted. The advertisement could say "you are seeing this because we think you're a female aged 18-30" or "you are seeing this because you previously bought a new car." This would alleviate the fear of the unknown and put power back in the hands of consumers, because we would know when and how predictions are made about us.
Once consumers see how companies use their data to make predictions, they are able to decide whether the trade-off between loss of privacy versus personalised digital services is one that they're happy with. The future we do not want may already be here, or it may not, but it's hard to know right now because individuals do not know when or how organisations use their data to make decisions about them.
Dr David Stillwell spoke on this topic at this year's Cambridge Festival of Ideas on Saturday 24 October: http://www.festivalofideas.cam.ac.uk/