Computer users have long complained that they are drowning in data but starved of information. Enter data warehousing, a system that scans vast quantities of information to look for patterns that can't be found manually.
When selling air tickets, you only get one shot - after the flight has gone, there is nothing left to offer. It's not like being able to sell off old inventory cut-price. So airlines need to keep a very close check on customer preferences. If people start abandoning Monday morning's 9am flight from Manchester to London in favour of the plane that leaves an hour earlier, the airline has to respond fast to meet the changing demand before the competition. As with flight times, so with shifts in favour of Continental breakfasts, discounted seats or follow-on tickets to Paris.
Of course, airline systems have long been logging this type of information; it's just that for an equally lengthy period of time, sales and marketing staff have found it extremely difficult to access. Not if they work at British Airways (BA), however. Five years ago, the company grasped the nettle and took its first steps towards creating a data warehouse, a computer system that scans vast quantities of information to look for patterns or correspondences that would be almost impossible to find manually.
The BA system contains all the latest sales and booking information for the company's 1,000 flights a day to more than 170 destinations. The 360-gigabyte database is equivalent to some 180 million A4 pages of text.
Huge as it is, sales and marketing staff can investigate this data at the touch of a few buttons, slicing it whichever way they want to spot hidden opportunities and identify potentially lucrative new markets. The data warehouse has proved so successful that it is being tripled in size during the next few months to encompass market share information and industry average performance statistics so that BA can better measure itself against the opposition.
'Suppose the sales manager for a particular route wants to target families,' says Peter Blundell, BA's data design group manager, 'then the system can show at what times infants travel, whether this tends to be on working days, if there is a correlation with the school holidays, and what the demand for smoking seats is on those flights.' The manager can then design an optimal price and schedule structure to appeal to the target families.
Computer users have long complained that they are drowning in data but starved of information. And over the years the software industry has come up with a series of statistical analysis tools to help. But these were designed to identify specific trends in the data. Programmers knew what they were looking for before they wrote the computer coding. Data warehouses, in contrast, are built to cope with ad hoc queries and to spot hitherto unknown correspondences such as loss-making product lines or most profitable customers. They can be used to examine historical corporate data going back many years and are a very effective way of deploying IT for competitive advantage.
Retailers have been among the data warehouse pioneers. The US supermarket chain Wal-Mart achieved early success when it spotted a link between sales of beer and babies' nappies on Friday evenings. By placing the two side by side in its stores, the company encouraged even more fathers to take home an extra six-pack with their babies' toiletries after work. Similarly in the UK, Woolworths is using a data warehouse to track what happens to the 40,000 products it sells throughout its 800 stores. The system, which cost an initial £2.5 million, is claimed to have boosted its sales in women's toiletries alone last Christmas by more than £8 million.
Hoteliers, meanwhile, are using data warehouses to track customer preferences for rooms with showers, flowers or extra pillows. They can then ensure that regular clients get their favourite type of room, and adjust pricing structures accordingly. For their part, manufacturers such as HP Bulmer, the Hereford-based cider company, are deploying data warehouses to monitor distribution, sales and the profitability of individual products.
Financial institutions also adopted the technology early on. Capital One, the US credit card company, has segmented the marketplace so minutely by means of a data warehouse that it has been able to issue not three or four types of credit card but hundreds. In the UK, Churchill Insurance is using a network of data warehouses to meet its goal of providing every caller with a quote in under three minutes. Meanwhile Reuters has just gone live with a massive system containing up to 10 years' history of prices on almost every financial instrument, including equities and options, as well as company information, money rates and third party data.
Data warehousing is expected to be one of the computer industry's biggest growth markets during the next few years. Sales forecasts vary from £1.3 billion to £5 billion worldwide within two years. In the UK, the market will grow from £82 million this year to £100 million in 1997, according to market research company Data Warehouse Network.
Yet, as with many computer innovations, data warehouses suffer from over-ambitious claims by suppliers. 'The hype in the market is overwhelming compared with people's capabilities when they are starting out,' says Ken Nicolson, director of marketing at California-based Red Brick Systems, which specialises in data warehousing.
Already there is a significant amount of disenchantment. It often begins when new adopters learn how expensive data warehouses can be. The average price according to industry surveys is £2 million to £3 million, but some companies say they have spent up to 10 times that amount setting up their systems. Reuters, for one, has forked out more than £65 million. Moreover, much of the set-up cost may involve the duplication of databases already held on other systems - a tough one to take to the board.
Warehouses also incur substantial training costs. While ultimately data warehouses will be easy enough for the most technophobic of managers, in the meantime the tools and commands can be difficult to master. 'The most significant cost associated with data warehousing is not the system but the people,' says BA's Blundell. His view is echoed by Donal O'Shea, data warehouse consultant at CSC Index: 'Current systems are so difficult that only about 10% of users ever wind up being expert,' he says.
International Data Corporation, the US market research company, reckons the average return on investment from implementing a data warehouse is more than 400% over three years. Yet many users - Data Warehouse Network puts the number at 64% - do not even try to cost-justify their data warehouses before implementation. One reason for this is the inability to know in advance exactly how the system will be used. 'People can't and shouldn't try to predict how it will be used, because if the implementation is a good one, usage will be far more extensive than anybody imagined,' says O'Shea.
Even after implementation, the cost benefits are hard to quantify because data warehouses tend to help with softer business activities such as marketing. 'The data warehouse is not like an operational system that will save manpower,' says Blundell. BA is convinced its system is playing a valuable role in maintaining the company's position as the world's most profitable airline, and spends several million pounds a year on it. 'But measuring the cost benefits would be impossible,' Blundell claims.
Much of the installation cost is caused by the need to overhaul existing data before it can be stored in the warehouse. Most organisations with multiple databases have inconsistencies, for example in employee addresses, customer identities, or product and service definitions. Ironing these out is expensive. 'The major focus at BA over the last five years has been getting the data quality and the data definitions right,' elaborates Blundell. The advantage is that once the task is complete, it need not be repeated, and a clean, consistent set of data has been created. This eliminates a huge amount of time-wasting in meetings right up to board level. As Martin Dixon, a BA general manager, says, 'At least nobody need waste their time arguing about whether the basic figures are right'.
Over-ambition is the biggest problem for those setting up data warehouses.
'Most companies try to bite off more than they can chew and tackle tasks that are far too complex,' says Red Brick's Nicolson. He often gets called in to 'nightmare situations' where companies have spent vast sums on data warehouses only to find they don't work. 'In most cases, this is because the project is too big and unmanageable,' Nicolson says. Firms sometimes try to kick off using all the data they have accumulated in every department for the past five years. 'Making this kind of change is like using pencil and paper one day, and automating your entire business the next,' says Nicolson.
The likelihood that individual users will require different views of data makes large data warehouses difficult to create from scratch. A sales manager might want weekly updates, for example, whereas a financial director with a global view might only need monthly summaries. Unless it is blessed with an enormous amount of processing power, it is hard to optimise the system to both types of needs.
A better approach is to subdivide the data warehouse, or to start off with smaller, isolated projects, beginning perhaps with an area of the business that has most to gain from the technology, or one where users are particularly enthusiastic.
Hughes Aircraft, for example, set up separate systems for manufacturing, financial analysis and human resources. These smaller data 'marts' can be technically compatible so that they can be combined at a later stage.
'Think big, then start small,' advises Blundell.
BA's data warehouse has reached the point where it can provide summary data for senior executives, detailed information on flights for individual route managers, and an even greater level of analysis for the company's researchers. 'Its strength is in being able to use the same system to answer all these questions,' says Dixon. Previously, investigating campaign results, market trends or the effects of currency fluctuations meant switching between numerous local, central and specialised applications. But even BA does not plan to put all its corporate data into the warehouse. Data hypermarkets may be going too far, says Dixon.
'There is always some information that will only ever be of interest to a specialised audience.'
Data warehousing raises a number of cultural issues such as the problems which arise when people are not used to sharing their data. 'People need to open up access to information of which historically they might have been custodians, so that it is freely available to all who need it,' says Blundell. IT staff can also be a problem. They need to be able to produce demonstration systems quickly and to think themselves into the shoes of line management without detailed requirements specifications. It is no good relying on users to predict what they want because they are more likely to think in terms of the reports they currently generate, which are very different.
Procurement can also be fraught with hazards. The soaring popularity of data warehousing means every self-respecting IT supplier wants to offer it. 'The trouble is that the product they are currently selling as a data warehouse was probably sold as a client-server solution last year and a decision support system solution the year before,' Blundell says. Anything with a recycled feel is best avoided.
The good news, however, is that data warehouses are gradually becoming easier to use. Some companies are even using intranets (internal company networks modelled on Web technology) to provide access to their data warehouses via standard Internet browsers such as Netscape's Navigator or Microsoft's Explorer.
And data visualisation techniques currently under development are enabling bar graphs and pie charts to be replaced with scatter diagrams which can highlight the effect of making one change on everything else. 'When the tools mature, it will make things much easier for line managers to spot the patterns in data,' Blundell says.
Even then, data warehousing will have its limitations. You still need humans to put the information in context. Take the surge in traffic on BA's Atlanta routes this summer. To cope with the increased volumes, the regular DC10 has been replaced with a fleet of 747s. 'If we were to look at the figures in a darkened room, we might conclude that business is so fantastic on the Georgia routes that we will need even more planes next year,' says Blundell. But a human would immediately spot the link with the Olympics. Data warehouses cannot know everything.