Storage Toolkit

Too much information stores up problems

Rupert Goodwins ZDNet.co.uk

Published: 25 Mar 2004 17:05 GMT

There is an unnamed corollary to Moore's Law that while hard disk storage capacity is growing exponentially, the time taken to fill it is a constant. A year ago, I moved from a twenty-gig drive to an eighty: today, it is officially stuffed to the gunwales.

All my life, I've been battling with storage. While I'm still in my thirties the problem has been with us for nearly fifty years, ever since IBM built the 305 RAMAC in 1956. That first hard disk system used fifty 24-inch diameter platters to serve up five megabytes: a quick check on my system shows I have 2,653 files larger than that, and I don't know what most of them are. Progress.

Information technology isn't very good at applying technology to information. It's fantastically good at acquiring data and stuffing it into cubbyholes -- but information is more than bits and bytes. Information is data in context: lose the context, and all you've got is gibberish and a full hard disk.

The problem is universal and has funded Larry Ellison's toy habit for decades. We are trying to force the entire world, with all its complexities and rough edges, into numbers that follow rules and models that match reality. Those with a taste for Borges will feel right at home with this fascinating and important practical and philosophical problem, and as we move our society and economy into the digital age it will become ever more so. I can't wait to see what happens next.

Meanwhile, however, my disk is full.

It is full because I have wholeheartedly subscribed to my role in the digital diaspora and now subsist on a sensorial diet of bits. My physical music collection gathers dust, my video is streamed, my personal and business communications are mediated through email and online voicemail. The stuff comes in at a megabit a second, and it never goes away. To deal with it, I have to know what it is.

There is a classic solution to managing information, first proposed by Harvard psychologist George A Miller at around the same time that IBM was building the 305 RAMAC. Called chunking, it's the principle that information can only be managed or communicated -- the two tasks are intimately intertwined -- by breaking it down into small sets of contextually related ideas. Miller looked at short-term human memory, and deduced that people can cope with around seven ideas at once. More than that and they have to throw something away in making a decision. These days, we'd call it cache management. It is a powerful and effective concept, and it has been totally ignored by software designers.

So what is a data collector to do?