In the climactic scene of The Matrix, Keanu Reeves' character experiences a Damascene moment when the computer-generated world he inhabits is revealed for what it actually is - pure data - and his surroundings resolve themselves into falling streams of noughts and ones, a kind of binary rain. His ability to see and manipulate this data gives him the superhuman power to dodge bullets, run through walls and leap tall buildings in a single bound - you know the sort of thing.
Sounds far-fetched, but this is the kind of revelation that many in commerce and public life would love to share. Everyone in government and business wants more information about the people they need to influence, be they constituents, customers or competitors - how they behave, what they believe, where they go, how they spend their money and if it's legal. The answers are all in the data.
At least, that is what the companies that prepare and sell information would have us believe. And it is big business.
Multinationals will spend millions on research to identify new markets for everything from yogurt to TVs. Shareholders want to know every detail about the companies they invest in and the people who run them. Retailers want a larger 'share of wallet' - more of our money, in other words. The Government wants us to carry ID cards so that they can convince themselves that they know who we all are. And we'd all like to be sure that the hospital we are about to be admitted to has a good record of success in the treatment we are about to have.
But is it really that simple? Is the truth really in the numbers? What are the companies that want to know so much about us, and how concerned should we be about it?
If you've ever applied for a mortgage or a credit card, bought a pay-monthly mobile phone or taken advantage of an interest-free deal to buy a new sofa or hi-fi, then whether you know it or not, you've probably already had dealings with one of the biggest names in the data business: Experian. Its credit scoring systems have become the de facto high street standard - back in The Matrix, these are the black-suited, shades-wearing data cops: mess with them and you could get blacklisted.
The Nottingham-based firm has a long history, having started out as the credit-checking arm of catalogue sales giant GUS. From such inauspicious beginnings, deciding who was good for a few quid on a new toaster, a credit reference business worth £100 million has grown up, not to mention a bewildering array of hugely lucrative information-based spin-offs. 'Things really took off in the '80s. There was an information wave then, which we are still riding now,' says chief operating officer Richard Fiddis.
One of the cutest aspects of Experian's business - and that of many other data companies - is that the services it sells to clients are heavily reliant on information that it persuaded those same companies to give it free of charge in the first place. So banks, retailers and credit card companies all hand over the financial histories of their customers without a murmur. This neat trick is usually pulled off using a variation of the 'for the greater good' argument - in this case, that knowing whether a customer is a sound credit risk is a good thing all round, and can be done sufficiently accurately only if all the big players in the market pool their data.
But to concentrate too much on that aspect is to miss the point that raw data is just the beginning of the story, says Fiddis. It is processed, overlaid with information about the place you live in and finally tailored to suit the individual client's appetite for risk. 'We have 400 million credit records on our database, but the data itself is useless when it comes to making a quick decision,' explains Fiddis. 'So we have 100 PhDs working out credit-scoring methodologies to make it much easier.
'Now we also have application-processing systems that do the whole thing automatically. Mortgage lenders using the system hardly ever look at the raw data; they just see our product, which makes the decision for them.'
This sense that we are what's on the database and that as individuals we are powerless to stop ourselves being defined and bracketed by our behaviour as consumers is one of the sector's most controversial aspects.
'People in the data business somehow believe that data is reality,' says Ian Angell, professor of information systems at the London School of Economics.
'In fact, they believe that what's on the database is more real than reality.
They have tidy minds and they work on the auditing principle. But life isn't like that - if the auditing principle worked, we wouldn't have Enron or Parmalat.'
And once something is on the system, it is very hard to get it off. Angell continues: 'The big question here is: who owns the data about you? I don't have any store or loyalty cards because I find them offensive. If I don't want to have data about me collected, I shouldn't have to opt out. And if I do want to sell my personal information, it should be for a decent rate, not for the paltry sums on offer now.'
At the moment, it seems that only crooks have put this kind of individual price ticket on our personal data. Identity theft is a big and growing problem, reckoned to cost about £1.3 billion a year in the UK alone. Dustbin scavengers in London can expect to receive up to a tenner from identity fraudsters for a piece of mail with a full name and address on it, and they might get up to £30 for more detailed records. The crooks then apply for credit cards, bank accounts and loans fraudulently, using the information to fool the company concerned. The first thing the victim knows about it is usually when their card is destroyed at the checkout or a credit application is refused.
And credit checking is only one seam in the data goldmine. Experian itself has 13 divisions, which mine a remarkably diverse range - everything from insurance fraud prevention to economic forecasting. But there are plenty of other rich pickings to be had. As our three profiles demonstrate, there's good money to be made slicing and dicing the facts and figures in markets as diverse as company directors, the health service and high-street shoppers.
The next big step will be predictive analytics, says Noel Coelho, UK general manager for statistical analysis software firm SPSS. 'Initiatives like customer relationship management have enabled companies to say: "Well, last month we lost 10,000 customers." The next step is to use data to predict which customers are most likely to leave you next month.'
To do that, companies like SPSS take a firm's data - call records for a phone company or account histories for a credit card business - add in qualitative and quantitative information, and subject the whole lot to some heavy-duty statistical theory. 'Center Parcs used our systems to cut its mailing costs by 50% and increase revenues by Eu1.5 million.'
Of course, the thirst for knowledge is a natural human desire, without which we would still be living in caves. What is significant about the current state of our obsession with information is that computer technology is now so advanced we can indulge it to an unprecedented extent. Records for most UK citizens are estimated to be held on more than 700 separate databases, with more to come. And despite data protection regulation, most of us have no idea what is held where and by whom.
Even some people in the industry believe that the balance of power needs to be shifted. 'Credit reference companies have amassed huge amounts of data on us - vastly more than there was 15 years ago,' say Ian Liddicoat, managing director for Europe, the Middle East and Africa at marketing strategy consultant MDS Global Consulting. 'Most people have no idea that their file includes all their mortgage details, for example. This should be a legitimate concern for consumers.'
There has already been a backlash against excessive consumer data collection, says Liddicoat, and more people now read the small print and tick the opt-out box. He predicts that the European market will go the same way as the US, where consumers are much more vocal in their demands for transparency and accountability. Over the coming years, he says, 'credit reference agencies will be pushed more and more into the public eye. They will find the experience very uncomfortable.'
But the data business is here to stay and it is doing good work. 'At best, these businesses are a benefit,' concedes Angell. A sensible and rigorous approach to data can be hugely beneficial in improving the performance of public services, as well as helping to sell more shampoo, toothpaste and deodorant to 35-year-old professionals who drive BMWs.
Perhaps our best defence against the science fiction of The Matrix becoming reality lies in old-fashioned human nature, suggests Angell, for the truth is that we are unpredictable creatures. 'Human complexity is our best defence against this kind of thing. It will be fascinating to watch the effects of human complexity on the data business over the next 10 years.'
When Tim Kelsey and his pregnant wife tried to find the best hospital for their child to be born in a few years ago, they soon ran up against one of the NHS's biggest problems - information. Although they had a choice of local maternity units, they had no information about how these were operated, what type of treatment they offered or how they were performing relative to one another. 'We couldn't find out what we wanted to know, so we picked one at random. It wasn't a good experience.'
Out of this personal trauma was born Dr Foster, a research organisation producing data on the quality of healthcare in Britain, whose first effort was the 2001 Good Hospital Guide. Published in the Sunday Times - where Kelsey was news editor at the time - the guide listed headline clinical performance figures for every NHS trust in England. It was based on ground-breaking research undertaken by Professor Sir Brian Jarman at Imperial College. 'I later discovered it was a world first,' says Kelsey.
What was intended as a user's guide created huge interest within the NHS, too, and the Kelseys realised it wasn't just patients-to-be who were crying out for more data. 'We walked into this great market opportunity. There is very little data that health service managers can use to measure their performance, or that taxpayers can use to hold their health service to account.'
That was back in 2001. Now Kelsey has quit Fleet Street and is full-time chief executive of Dr Foster, which has doubled its turnover annually for the past two years and generated revenues of £2.5 million in 2003.
The heart of the company - which now operates as a business-to-business information provider, selling clinical outcomes performance data to hospitals and NHS trusts across the UK - is information that the NHS collects but doesn't have resources to analyse. The details of 12 million in-patient episodes annually are routinely recorded. 'We've developed techniques to make this data useful, to help answer questions like "Do we need a new cardiac ward?". Lives will be saved as a result of hospitals using our tools.'
Hospital performance depends on the social profile of the catchment area, and Kelsey is keen to stress the academic rigour of the comparison process.
After all, a schools league table-type controversy could be bad for business.
'Our biggest expense is the Dr Foster unit at Imperial College, where the methodologies we use to adjust our results for things like age and sex are developed. We have an ethics committee and we're very transparent. We say the NHS must be accountable, so it's vital we are too.'
Of all the many loyalty cards now available to the British consumer, only one can be used in more than one shop - the Nectar Card, run by Loyalty Management UK (the firm behind the favourite loyalty programme of frequent fliers, Air Miles). Nectar's list of participating retailers stands at 11 and includes Sainsbury's (out of whose Reward Card scheme Nectar was developed), BP, Barclaycard and Vodafone.
This, says Nectar CEO Rob Gierkink, is one of the secrets of its success.
Not only does the 'multiple sponsor' approach make for a more attractive offer for consumers, but, crucially, the sponsors do better as well. 'The more sponsors we have, the more customer spend goes up. People who collect points at two, three or four sponsors spend more with each of them than those who collect at only one.'
Of course, the real value in loyalty schemes - so ably demonstrated by the daddy of them all, the Tesco Clubcard - is in the goldmine of marketing data they provide about spending patterns. Here, claims Gierkink, Nectar's breadth of coverage pays dividends too. 'Our range of sponsors allows us to develop a fuller picture of a person's habits. Tesco can't compete with this, or our cheap mail.'
That cheap mail is a Nectar speciality, and addresses one of the biggest issues facing loyalty marketing schemes - money spent on postage eats away at the budget for loyalty rewards. 'It costs us only 1p to get an offer to a customer. We send them out with our Nectar points statements,' explains Gierkink. The coupons offer a range of discounts tailored to the profile of each customer.
Typically, customers get one Nectar point for every £2 spent, and Gierkink claims that the 'average Nectar family' earns 18,000 points a year. That's £90 off the bill at Sainsbury's, nine pairs of cinema tickets or three flights to Amsterdam. At the time of writing, sponsor Ford was offering a whopping 12,000 points to any Nectar user buying a diesel-engined car.
The latest innovation is tackling people who've just moved home, sending them a map of their new area marked with the nearest stores of all the Nectar sponsors. 'It's mass-customised marketing, triggered by a specific event,' says Gierkink proudly.
Nectar now has about 13.5 million people signed up, with 10 million of those described as active collectors. 'We activate 50,000 sign-up packs a week,' he claims. So which is biggest - Clubcard or Nectar? 'We have friendly banter with Tesco over which scheme is largest.'