Data storage is about as exciting as plumbing, but just as necessary. In both cases, you notice them only when they spring a leak or something gets blocked. Even IT people find storage boring. They'd rather build fancy networks and clever software than worry about disk drives and tape decks, so it is little wonder the subject is given scant attention.
And who cares, anyway? We all know computer hardware is dirt-cheap, and these days $500 (or £300) will buy you a disk drive with a capacity of 200 gigabytes, so need we worry? Well, yes and no. The unit cost of storage is coming down, but not fast enough to cancel out our rising needs. Says Paul Brown, a storage specialist at IBM: 'The cost per gigabyte of disk capacity is going down year on year, but the overall operational cost of managing a storage environment will go up. Some 80% of this is operational staff cost. With more data, you need more resources to look after the growing storage infrastructure.'
The sheer volume of data we keep is beginning to cause problems, and we have to learn to manage it all. Well-managed data is vital in running an efficient business but also essential for compliance with a wide range of regulations, from corporate governance to data protection and freedom of information. So like it or not, we can't ignore storage any more. 1. YOUR BULGING INBOX
Although we still get tons of mail through the post that has to be processed (there might be a cheque in one of those envelopes), our e-mail inboxes overflow not just with messages but with pictures, video clips, charts and spreadsheets. It's estimated that our storage needs are rising by 60%-70% a year as getting bigger and we store more of them. When your desktop PC had a 10Mb hard disk, that put a lid on what you could store. Now that machines can hold many gigabytes, there's less incentive to have a clear-out. Another major factor is regulation, which compels organisations to keep more of their records for ever-longer periods.
2. MORE KINDS OF DATA
There are now loads of different types of information to be stored. For a start, many firms now scan and digitise incoming snail mail. CCTV security systems are also going digital, storing their images on a disk rather than on a VHS tape. With internet telephony, voice conversations are digital too, and can be easily recorded in case of subsequent query or dispute. In retail, transport and distribution, firm are adopting the RFID (radio-frequency identifier) tag to track goods in transit, on the shelves and through point-of-sale. According to IBM's Brown, this will drive significant data growth because RFID tags can hold more details than barcodes and allow tracking of goods down to individual stock items. 'This could put a strain on storage infrastructures if they are not carefully managed,' he warns.
3. EMPTY DISKS
Paradoxically, most of the disk drives sitting on PCs and servers are half-empty - their capacity is locked into the system. 'Eighty per cent of servers are not overloaded at all,' says Hamish MacArthur of storage consultancy Macarthur Stroud International. 'It's the 20% that are 110% loaded that cause all the problems.' A lot of data is 'unofficial', and duplicated too. 'Someone downloads the latest Kylie video and sends it to everyone else, so they all have a copy.' Overcome this wastage by concentrating the data into one place as network-attached storage (NAS) or a storage area network (SAN), rather than having it siloed in different systems. This brings huge savings, and making the best use of available capacity simplifies data management.
As for the Kylie videos, Jon Pavitt at Unisys, says: 'Just announce that you're carrying out an audit of the kind of information being held on company storage. You'd be amazed at how much will disappear.'
4. MANAGEMENT'S CALL
The IT department will decide on how to store the data - on disk, on tape or on optical disks - and where. But their decisions have to match to the needs of the business, its appetite for risk and the requirements of its regulators. Discussion on how information is stored and safeguarded must involve business line managers, board members, HR and the legal/compliance department. It may be cheap to store your archives offshore in Sri Lanka, say, but is that allowed under the Data Protection Act or the rules operating in your industry? How much data should you retain, and for how long? What can you throw away? Again, the legal eagles and auditors will take a view here. Line managers opting to save everything need to know from IT the cost of doing so. Cost-cutters may opt to delete everything after 30 days, but that decision may well need to be approved by compliance or auditing staff.
5. SAFETY BACK-UP
It's fine having your entire company's data neatly digitised, but what if the technology goes wrong? Significant data loss is likely to put a company out of business. Regular back-ups allow you to revert to a copy of your files and carry on with minimum disruption. But last May, a report from the Chartered Management Institute revealed that only 49% of firms have a business continuity plan. And of those, only 37% actually rehearsed it. And guess what? - four out of five rehearsals unncovered flaws in the plan that needed to be put right. Observes Pavitt at Unisys: 'The problem with rapid data growth is that companies don't have the resource within their management teams to cope with it. So they start taking short cuts, thereby putting risk into the business. Back-up becomes a problem, because firms are going global and trading 24/7, so there is no back-up window.'
6. SECURITY SENSE
Underlying an information security programme are the letters CIA - for confidentiality, integrity and availability - and this applies especially to data. The C means you don't let unauthorised people see it; the I means the data is not altered or destroyed without permission; and A means you can get at it when you need it.Satisfying those three basic needs is not easy. As red-faced execs of the Nationwide Building Society admitted in November, a laptop burgled from an employee's home had contained customer details that could be read by the thief. They certainly broke the C rule, and allowed too much A. With memory sticks now capable of holding 16Gb, it's easy to siphon off the equivalent of a lorry-load of printed material and slip it into your pocket. Some people even use an iPod to steal data. Protection of data must be a major concern, especially because of the duty of care for personal and financial information.
7. THE NEED TO COMPLY
A raft of new regulation, from Sarbanes-Oxley in the US to Basel II and MiFID in Europe, is forcing companies to store more data for longer. You may be required to store documents, e-mails, voice calls and other forms of communications for several years. Not only that, you need to be able to retrieve the information at a speed acceptable to the regulator.Then you must prove the data has not been tampered with; that requires detailed system logs.
8. THE NEED TO CATEGORISE
Governments and the military have always classified information according to its importance and level of confidentiality. Now organisations need to do the same. Business managers should work with IT to decide which information is vital and which can be deleted or consigned to long-term archiving. Una du Noyer of services company Capgemini says one of her customers moves any data not accessed for six months off the main systems and on to cheap storage. 'They found that at least 50% of information was never accessed. A crude dating system is a broad-brush approach, but it can reap benefits.'
9. WHERE TO STORE?
Information can be stored on disk drives (instant access) or on tapes (slower but cheaper per megabyte stored). Archive data may also be stored on optical disks such as CDs. The differential costs between these media inspired IT to develop information lifecycle management (ILM), which tries to assign data to different types of storage according to its age and importance. Data for processing will be on a disk attached to the main network or servers, while aged data can be shifted on to a slower, cheaper medium such as tape or optical disk. Some say switching between media isn't worth the effort and advise using the fastest medium you can afford in order to cut operating costs.
10. WHAT TO OUTSOURCE
Faced with the burden and dangers of storage management, some firms offload the task on to a services company. Pavitt of Unisys urges a careful approach: 'Some companies like the idea of outsourcing the responsibility, but they don't like the idea of having their business assets co-hosted with their competitors. It is a psychological resistance.' Firms that have their data properly classified, he says, can outsource the management of non-critical data but retain the management of what is crucial or confidential. Compliance officers may block a move to outsource management of personal data, for example, especially offshore.