A good business intelligence strategy will flow from the strategic objectives of the company itself. It will help to determine the key performance indicators for the business and, through the generation of balanced scorecards, show people at all levels how well they are doing.
Ever since the 1970s, IT departments have been trying to produce useful and accurate information for the rest of the business. The long road toimproving that process has been arduous and not always successful.
The first Executive Information Systems (EIS) of the late 1970s and early '80s were grand and expensive affairs. They usually consisted of a purpose-built room decked out like a mini cinema, with a large screen and a few buttons for the chief executive to press. The systems allowed the senior board to analyse profit margins, stock levels, productivity and whatever else they asked for. They made some use of graphics and, on the whole, gave senior management the impression they had their fingers on the pulse of the company.
But beneath the sleek control panel was a more mundane reality. The information for EIS systems had to be gathered by hand by teams of people trawling through various systems around the company, and assembled piece by piece into a single set of information for the board of directors to view and analyse. It was a labour-intensive, error-prone process based on out-of-date information, and therefore of limited value.
EIS was soon succeeded by a series of newer generations of systems that promised to do the same job better and more cheaply - decision-support systems, data warehouses, online analysis and reporting systems, and various other bits of terminology, each offering just a little more by way of function and sophistication. But they were often hamstrung by the inability to access accurate and up-to-date data from around the organisation. By the time the business of assembling the data was complete, it was out of date. Like driving a car via the rear-view mirror, it could tell you where you had been but not where you were going.
At the same time, the world moved forward from mainframe computers to minicomputers to PCs and the connected world of the internet, expanding the number of people with a computer connection. And computer networks supported the corporate infrastructure, enabling data to flow more easily from one part of the business to another. The scene was set by the mid-1990s for a new breed of software tool going under the umbrella term of business intelligence (BI).
New terms entered the management lexicon, such as 'data cubes', 'slice and dice' and 'drill down'. The data cube is a subset of the company data, such as sales figures or staff turnover information. Each cube could be specially generated for a department or individual, which they could then analyse from different points of views, such as by month or by region (slice and dice), or examine in more detail (drill down). The software to do this required little training for the user, and gave them the chance to examine the data from various angles and make their own judgements.
And yet, despite improvements in networks, computer technology and the business intelligence tools themselves, the success rate of BI projects remains doggedly low. Analyst firm Gartner estimates that about half of all BI projects fail to deliver on their promise of giving accurate information to users in a timely fashion, or are abandoned completely. So what is the big problem? One of the greatest obstacles is the same as 20 or 30 years ago - a lack of access to good data.
"Most companies still have fundamental problems with data quality," says Eddie Short, head of the business intelligence practice at consultancy Capgemini. "They still rekey information using data from untrusted sources and multiple customer databases with inconsistent information." He says even something as simple as having a customer recorded under variants of his name in different parts of the organisation (J Smith, John Smith, JR Smith) can upset any attempt to build an overview of what is going on.
The goal of a single coherent view of the data is further frustrated by the widespread use of PC spreadsheets (usually Microsoft Excel) for making business decisions. It is estimated that Excel is installed on more than 300 million desktop PCs around the world, and it has become the favourite tool of managers who want to do their own financial modelling, budgeting and analysis of figures. If all those spreadsheets took their data from centrally controlled data repositories of company information, it might be less of a problem. But most of them exist in isolation, with data keyed in by the managers themselves and a good proportion containing programming errors. Anyone who doubts the seriousness of the problem should consult the long list of real-life disaster stories at www.eusprig.com.
The appeal of the spreadsheet is obvious, though. It is easy to use and it allows users to view the data in the way they need, and to extract the kind of information they want, instead of having to wait for an IT department to supply it. "We have fortunes invested in IT, but executives still have their own trusted version of the truth on an Excel spreadsheet," says Short.
Fortunately, the big BI system vendors, such as SAP, Oracle, Business Objects and Cognos, have started to tackle this problem by introducing extra tools to help companies cleanse their data and iron out differences between systems. They have even introduced 'Export to Excel' features into their systems, making it easier for users to pull live data from the central BI system, rather than rekeying it by hand.
But it is still a huge task, as data volumes are constantly increasing. According to a recent survey of 100 UK companies, for instance, a third of them took a year or more to sort out the quality of their data. That seems par for the course, even in the US where adoption of BI is reckoned to be more advanced than in Europe. But according to Michael Corcoran, head of marketing for Information Builders in New York, it is a mistake to delay a BI implementation. "People believe they have to wait until they fix the data quality problem before they start doing BI," he says. "But you don't really have an idea of your data quality until you start using BI. When businesspeople have access to the information, they will spot the inconsistencies better than anyone else."
The one effective way to build what every reliable BI system needs - a single version of the truth - is to create a data warehouse, a central repository that is fed by the various systems of the company (sales, accounting, stock, cutomers) and reflects a definitive picture of what is happening. With all the data in a single place, it is then possible to start exploiting it properly for business benefits by allowing it to be used for decision-making throughout the organisation. Yet companies have been slow to reach even this stage of the process. The same survey showed that only half have initiated data warehousing projects. Nearly all said the quality of their data was poor and that the purpose of building a data warehouse was to rectify the situation.
But having the data properly organised is only half the job. It's what you do with the data that matters. The four stages to enlightenment are data, information, insight and intelligence. The raw data by itself is of little value, but when it is gathered all together then information can be derived. Insight comes, for example, from being able to compare sales of different product lines in different parts of the world, or at different times of the year. The intelligent organisation manages to gather and analyse the data, so that informed decisions can be made at all levels of the organisation, from the chief executive down to the humblest worker, allowing them to make good decisions based on accurate information.
UK supermarket chain Tesco is a good example. During the early 1990s, it began a loyalty card programme that enabled it to gather vast amounts of data about customers and their buying habits. Sophisticated analysis of that information helped the company management overhaul the competition in its home market, and support rapid growth across Europe, the Far East and soon into the US. The same data is also used to drive information screens at store level, so that staff can see how they are performing against their key performance indicators (stock availability, quality of service).
In the best BI systems, the information is tailored to the needs of each group of users. The company board may want big-picture summaries, showing how they are meeting their key performance indicators (KPIs), while the front-line worker may need some very specific information packaged in a way that will support what they are doing.
Alison Whitby, a BI specialist with consultancy Morse, gives an example of a telephone ordering line. "You might have a junior person taking an order, and the system flags up that the customer has not paid their last invoice. Instead of blocking the order, the system will check to see if the customer is a big spender or likely to cause problems. It may advise that the worker asks what has happened to the payment and will even allow them to call up details of the order for the customer's information. It means the worker can take an informed decision on whether to take the order."
In other words, access to live data about the customer and their order history provides the worker with the right information to deal properly with the customer, rather than potentially antagonising a good client. Whitby says that this extended use of BI is what marks out the best exponents from less successful implementations. Her company has even developed exploitation benchmarks for different industries, so that companies can measure themselves against their competitors. She says that the factors for success include the number of users (the more the better), the number of functional areas where BI is used, how far the information is available throughout the company, and how well it is linked to corporate goals.
A good BI strategy will flow from the strategic objectives of the company itself. It will help to determine the key performance indicators for the business and, through the generation of balanced scorecards, show people at all levels how well they are doing. "Leaders use BI to drive KPIs, particularly beyond just reporting and analysing functions. The best users move it out beyond the analysts and controllers, right down to the front-line workers," says Whitby.
Gerry Brown, an analyst with Bloor Research, agrees: "You really need to start getting metrics into everybody's job description. It also ensures that the measures on the shop floor are aligned to the business strategy. It means you can introduce strategies more quickly into an organisation, to turn the tanker much faster, if everybody's metrics reflect the wishes of where the CEO wants the company to go, instead of waiting a couple of years before they get the hang of what he wants."
It all sounds so easy, but companies with that kind of integrated approach still represent a small minority. Morse's Whitby says: "The biggest barrier is the failure of management to put enough priority and thinking into it. They fail to realise that a BI strategy is crucial for them. Therefore, they don't sponsor it or put their weight behind it. They forget that BI is a core competence of their business."
Capgemini's Short identifies a deeper problem. "You need the decision support systems in place, but also a transformation of the culture of decision-making. It's about moving decision-making from the heart or the gut, to the head." He says many European managers take pride in being functionally innumerate and leaving it to the accountants. "But they are kidding themselves. Smart companies are much more analytical in their behaviour. The flipside of that is that when you have an intelligent organisation, there is a transparency of information. You need a much higher level of trust between executives, managers and staff. You need a culture that empowers people to take decisions. Companies that get there achieve a big change in business performance."
He says Tesco and Wal-Mart have both exploited the power of data warehouses and BI to make smart decisions. CapitalOne uses BI extensively to analyse customer habits and make a success of sub-prime lending, when so many others have incurred bad debt. "CapitalOne conducts about 30,000 'experiments' a year, polling its customer base to track its behaviour. It can weed out the defaulters and offer a sensible rate to others. It is smarter at analysing its customers."
In the field of manufacturing, General Electric ended up changing its whole business model based on the intelligence it gained from metering activity on its aircraft engines. The scheme was intended initially to predict more accurately when maintenance would be required and to maximise time in the air through intelligent diagnostics. The information showed GE that it would be more profitable to offer its engines not as a product, but as a service, paid for by actual flying hours.
"In the leading organisations, for the senior management, it's like being in the cockpit. They can see the key metrics of their organisation in terms of red/amber/green, or dials and graphs," says Short. "GE's vice-presidents have a fully up-to-date view - they can see the dashboard and drill down into the detail, or get on the phone to the person responsible. The global top 100 companies have that level of control, but outside that it is a small percentage."
He says the best users of the technology progress from historical analysis of data to predicting what will happen. "If you are an executive today, you have to make more decisions now, and faster," he says. "From the prime minister downwards, you need instant access to information. A few years ago, you could take several days to do the research and then respond in response to a crisis. You need more confidence and trust in the information at your disposal - but the reality is that most don't have that trust. So they make the wrong decisions."
A well-oiled business intelligence system is an essential part of modern business. Smart handling of information is what marks out the best companies, and as international competition mounts, the ability to make good decisions instantly will become even more important.
An increasingly popular approach is the idea of a special department, known as the BI Competence Centre (BICC), devoted to the production and analysis of data for the whole organisation. According to Paul Hulford, a senior manager at Cognos, a vendor of BI systems, more than half the companies he deals with are setting up such centres.
He says the main drivers are a desire to standardise and save costs, and to help rationalise the information held around the company. "It helps to get teams to communicate, develop best practice and come together to streamline their activities. And with scorecarding, it helps articulate which metrics should be used to measure against which strategic goals. That helps communicate the goals down through the organisation."
One example of such an organisation is utility Eneco Energie in Belgium, a company of some 4,500 employees. The company established its Business Intelligence Centre Excellence in September 2005, under the leadership of Ton van den Dung. He leads a team of six people and has close links with more than 20 other BI specialists scattered around various departments of the company. Between them, they work as the main contractor for all BI activity. "We negotiate and help to define the demand side of the information [the business], and then we translate it to the supply side [IT] to make sure that they match what is needed," says van den Dung. "The BICC plays the role of broker in the middle of that information."
It has also delivered benefits at an operational level by enabling the marketing department to segment its 2 million customers more accurately. Before the use of BI, a mailshot to all customers would yield a conversion rate of 10%-15% if they were lucky. Now, the more targeted campaigns, aimed at groups of 150,000 to 250,000 customers, give conversion rates of 85%, he says.
Van den Dung has two reporting lines, one to the CIO and the other to the CFO. The dual role is important to him: "My tip for anyone starting a BICC is to make sure that everything you do is from a business point of view."
SEARCH - THE NEXT FRONTIER
Everyone knows Google, the internet search engine, and how easy it is to use. All you do is type in your key words and up come thousands of results, with the most relevant at the top. But could this ease of use be applied within business to search for relevant information?
Most BI vendors have made alliances with Google and other search engine manufacturers over the past year, and some early applications are being deployed. The reason is that while BI has focused on handling numeric transactional data and providing numerical results, a whole other world of unstructured data is sitting inside organisations - in emails, text documents and even taped voice conversations - accounting for 80% of a company's information.
A combination of traditional BI and more flexible search functions should enable users to gather a much better picture of events. Mike Lynch, founder of search company Autonomy, says: "Most of the big sins are in the unstructured information. It's the 80% that hasn't been handled by BI, and it's the 80% that's likely to get you sued."
His own software was originally developed for the intelligence community, which needs to deal with masses of documents and phone calls, and to spot suspicious patterns. Banking and anti-fraud applications came next, and the software is also used now in healthcare, manufacturing and retail, where its ability to deal with probabilities rather than black-and-white decisions is increasingly valuable. Other BI vendors include FAST of Norway and Temis of France; Google also sells a search device that allows companies to search their own unstructured data.
However, Louella Fernandes, an analyst with Quocirca, says it will be a while before companies start using BI in any numbers. "There are higher priorities. Companies are still focusing on reporting on structured data and analytics."
But for a taster, Google is trialling a system that displays graphic information about companies' share-price performance, with an explanation why a share price has peaked or troughed at a certain point. It illustrates how hard data and text might sit together on a corporate dashboard.