When Child Support Agency chief executive Doug Smith faced a parliamentary select committee of MPs in November, he was asked if he could have done more to prevent a £456 million IT disaster. For disaster is what the committee had already branded the CSA's computer project earlier in the year, calling it 'defective' and saying that too many government IT schemes ended up as an 'appalling waste of public money'.
Smith was appearing before the Work and Pensions committee of the House of Commons to answer questions once again on the calamitous introduction of the new system, called CS2. Although the agency had received 478,000 applications for support in 18 months, payments from only 61,000 absent parents had been processed.
After a pensive moment, Smith said: 'We should have looked more carefully at the information we had been provided with around the turn of 2002-03 about the state of readiness of the IT system.' But though Smith was much derided by MPs, he had put his finger on one of the most common causes of IT disasters in the private and public sectors: credulity - sometimes termed 'irrational exuberance'.
It is a classic, because oft-repeated, scenario: a buyer puts too much faith in the ability of the supplier to deliver technology that will invigorate the business. In turn, the supplier's representatives put too much faith in their own abilities, and those of their clients, to understand the buyer's business processes. This mutual lack of understanding often leads to specifications that are unclear. In turn, this results in projects that keep changing their shape and direction, cost and timescale.
Such problems afflict IT projects undertaken by both the private and public sectors, but in the former, companies tend to learn from their mistakes. What sets public-sector projects apart is the repetition of failure, the size of the failures, and their public visibility. Officials running magistrates courts, for example, are now on their fourth attempt in 15 years to introduce national systems that will unify the administration of cases. The cost of the latest project has risen from an initial estimate of £150 million to £390 million, and there is no indication yet that a unified case management system will work.
The Department for Work and Pensions has also had years of learning lessons from IT disasters dating back to the 1980s. With such a vast store of knowledge behind it, it was surprising that this very department suffered in November what was described in the media as one of the most serious technology-based failures ever. A glitch in a routine upgrade of desktop operating systems led to tens of thousands of desktop systems going down for three days.
Some time after the incident, the department and its main IT suppliers, EDS and Microsoft, were still baffled as to exactly how such an everyday upgrade could go so wrong.
Typically, IT projects in the public sector begin with the prospect of legislation being passed that will require changes to existing systems. Or technology suppliers will lobby ministers and senior civil servants to adopt their advanced systems and IT-dependent ideas. For example, IT suppliers, in conjunction with the Passport Service and the Driver & Vehicle Licensing Agency, have been trying for many years to persuade ministers to introduce ID cards. According to home office documents, senior civil servants were working on ID cards in the late 1990s, even when the Home Secretary of the time, Jack Straw, was against the idea. Finally, David Blunkett acceded to the pressure.
Once a project is approved in principle, civil servants and consultants draw up strategic and outline business cases - documents that feature long lists of benefits that will more than justify the project's cost.
Only near the end of these documents are detailed risks of failure listed.
In an unpublished annex to the business case for the NHS's £6.2 billion IT programme, for example, these risks are mentioned only as a sideshow to the anticipated benefits. This is despite the fact that in the case of the NHS programme, a scoring of the risks gave the scheme 53 points out of a maximum of 72. This categorised the scheme as very high risk, but that did not stop the programme getting full ministerial approval. Now, two and a half years on, the scheme is running into trouble.
Once the business cases are approved - and sometimes long beforehand - officials begin work on drawing up specifications and plans for the procurement process. As an independent check that all is going well, the Treasury's Office of Government Commerce will carry out gateway reviews in which independent IT specialists assess the project at six critical stages. The first - called Gateway 0 - is a strategic assessment of the project's feasibility; the second (Gateway 1) is the business justification.
This is followed by assessments of the procurement strategy, investment decision, readiness for service and, finally, the benefits accrued from the operational system.
At these various stages, projects get a red, amber or green light. Red is a warning that the project should not proceed until weaknesses disclosed in a review are dealt with. Amber is cautionary, and green means there are no potentially serious problems.
The reviews are supposed to be mandatory for high-risk projects, but in practice they are voluntary: a report by the National Audit Office in 2004 found that departments selected which reviews they wanted to undertake, as if they were at the pick-'n'-mix counter in Woolworths.
The same report also found in some cases that departments chose the reviewers they wanted. And the reviews are always kept secret, so that Parliament, taxpayers and the media do not know when a project has shot through a red light, or a flawed system has been given a green one.
Another idea that works better on paper than it sometimes does in practice is the appointment of a senior responsible owner (SRO). This puts the same person in charge of a project all the way from conception to realising the benefits, thus avoiding the age-old problem of civil servants moving from job to job and not seeing a project through. But the audit office found that some departmental heads choose when they want to appoint an SRO; and that the NHS's IT programme has had three so far, two of whom have already departed.
Once the contract is awarded, excited project teams turn to delivering tangible benefits as soon as possible. The focus is on getting the right mix of hardware and software, installing the fastest networks, trying to limit changes to the specifications and designing testing plans.
By this stage, the cumulative weight of enthusiasm is so great that failure cannot be countenanced. Everyone working on a large, pioneering IT programme must have unwavering faith in the project's inevitable success, like the emperor's new clothes.
People can point out specific, solvable problems, but there is no place in the project for the dispassionate realist who questions whether anyone will like or want to use the new systems.
When, for example, Barry Kirwan, head of human factors at National Air Traffic Services, wrote a report pointing out that the screens planned for the new air traffic control centre in Swanwick, Hampshire, displayed text that was barely legible, his comments were not embraced with enthusiasm by his employer. 'There was resistance from our management to publishing the report internally,' said Kirwan.
But his warnings were later vindicated. When the system went live in 2002, five years late and £180 million over budget, air traffic controllers had difficulty reading some of the text on their screens. In one case, a controller treated a plane bound for Glasgow as if it were going to Cardiff after misreading the destination codes on his screen. In another, a controller misread the altitude of a plane and was about to send another into the wrong airspace.
Back on the ground at the CSA, the Work and Pensions committee has still not established what exactly went wrong with its IT project, despite an eight-month investigation. As for who should take the blame for Whitehall IT disasters - supplier or customer - this is rarely clear-cut. Usually, the fault lies with the buyer and its main supplier.
But why do public-sector contracts bomb so frequently? Any diligent study of the biggest failures will show that they are characterised by over-ambition, over-optimism and an unchecked belief by both customer and supplier that it is possible to introduce systems of enormous complexity quickly. Credulity is also a common element: suppliers tempt ministers with promises that the cosmic complexities of departmental affairs can be shrunk to the size of ordinary human understanding by the installation of new technology.
According to the Office of Government Commerce and the National Audit Office, the most common causes of IT failures on large publicly funded projects include a lack of skills in Whitehall to manage suppliers, and the lack of a clear link between the project and the organisation's key strategic priorities (see box). Another, unacknowleged, cause is that there is little incentive for senior civil servants to succeed, and little fear of failure. Rarely is anyone made responsible for an IT disaster, so there is no compelling need for departments and agencies to adhere to good practice and learn lessons from past mistakes.
In theory, ministers are responsible for any major failure in their department, but their incumbency tends to be transient and some of them are long gone by the time the contract ends in disappointment. Their successors can legitimately argue that they were not in post when the contract was signed; for example, the new Secretary of State for Work and Pensions, Alan Johnson, played no part in any of the department's high-profile IT failures.
So, with the civil service's culture of secrecy and habit of moving people around, it is difficult for Parliament to identify the culprit. It is also difficult for Parliament to find out exactly what caused a project to fail or to dispute the assertions of ministers and heads of departments when they seek to present disasters as successes. The Department of Health, for example, portrayed its national IT projects as successes until it launched a new £6.2 billion technology-based modernisation. At which point, it set about busily discrediting all the 'successful' IT projects that had gone before.
It is true that there are probably more cover-ups of IT disasters in the private sector than in Whitehall. But if systems fail at, say, Sainsbury and its shelves offer a poor selection of products, shoppers can go elsewhere.
The public has no alternative source of supply when the Criminal Records Bureau systems - a Capita project - can't process police checks on applicants for jobs working with vulnerable people and children; when there are delays issuing passports; or when students starting university don't receive their loans.
Yet there have been successes in the public sector. The Driver & Vehicle Licensing Agency has proved through repeated introductions of useful systems that being an 'intelligent customer' can make the difference between success and notoriety. Inland Revenue computers can run like a Swiss watch (prompting sceptics to say that the only Government IT projects that work are the ones designed to suck money out of our pockets). But these triumphs never seize the spotlight from the parade of failures.
The Work and Pensions committee, investigating why IT projects go wrong, concluded that the solution was not more books and guidance papers on project management, but a means of ensuring that departments adhere to good practice. Accountability and transparency are required here. Parliament needs to know how projects are, or are not, progressing. MPs can then hold departments and ministers to account before a drama turns into a crisis.
Meanwhile, the Government is embarking on some of the biggest and riskiest IT-related projects in the world. Tens of billions of pounds are being spent on ID cards, on the upgrading of welfare systems at the DWP, and on the NHS's IT-based reforms. Yet Parliament, the guardian of the interests of taxpayers, knows next to nothing about any of these projects.
The stage could be set for the disasters to continue as they have for more than 20 years, but this time on a scale not seen before.
WHY THINGS GO WRONG - Common reasons for public-sector IT failures
- No clear link to the organisation's key strategic priorities
- No clear senior management and ministerial ownership
- No effective engagement with stakeholders
- Lack of project management and risk management skills
- Failure to break down development and implementation into manageable
- Poor evaluation of proposals driven by initial price rather than
long-term value for money
- Poor understanding of and contact with the supply industry at senior
levels in the organisation
- No effective project team integration between client, the supplier
team and the supply chain
Source: Office of Government Commerce and National Audit Office.