Chris Anderson is the editor of Wired magazine, one of the bibles of the information age. His book, he tells us, 'began with a mistake'. He was talking to Robbie Vann-Adibe, the CEO of Ecast, a digital jukebox company. Digital jukeboxes are like their analogue counterparts but with one important difference: rather than holding, say, a hundred CDs, they have a broadband connection to the internet and patrons can choose from 10,000 music albums stored on a hard drive somewhere else.
Vann-Adibe had asked Anderson to guess what percentage of the 10,000 albums sold at least one track per quarter. Anderson guessed 50%. The answer was 98%. It was one of those moments that the novelist James Joyce called an 'epiphany' - a sudden spiritual manifestation that provides a blinding insight, in this case into the nature of the digital world.
Why is that figure of 98% so striking? Because it runs directly counter to one of the laws of retailing nature: that generally most of your profits come from a small percentage of your inventory. Anderson had the wit to realise that he had stumbled on something big, and he spent some time trying to figure out what was going on. He looked at lots of other online businesses to see if the Ecast phenomenon was repeated. It was.
Take Netflix, a company that rents out DVDs by post. It carries an inventory of 60,000 titles. How many of those are rented at least once on a typical day? The most common guesses are around 1,000, based on the idea that most customers will go for a small number of titles. The actual figure is between 35,000 and 40,000. Or to put it another way: every day, roughly two of every three movies that have ever been put onto DVD are rented by a Netflix customer. And every title in Netflix's inventory is rented out at least once a quarter. If you plot a graph of sales against items, you get a curve that starts high, slopes down quickly to near zero and then continues at that level for ever.
This means that a very small proportion of items sell in large numbers and everything else sells only in very small quantities. It's what statisticians call a 'power-law' distribution. But its striking visual feature is that it stretches out, apparently to infinity, on the right: it has a 'long tail' - which is where Anderson got the title of the Wired article in which he first wrote about the phenomenon.
It was, he found, the most popular and remarked-upon piece the magazine had ever published. Given that the power-law distribution is effectively a law of nature, why the excitement? Simply this: in the physical world, retailers can hold only a limited inventory. Shelf space costs money.
This means that they cannot hold stuff with a low probability of sale.
So they focus on items that have a chance of earning their keep.
By the same token, movie studios, record companies and publishers focus on trying to produce 'hits' - items on the magical left-hand edge of the power-law distribution. So we wind up with a system that is obsessed with hits - the world of the blockbuster.
Anderson realised that this system is a product of a world of scarce resources - such as shelf space. If you can relax that constraint, a parallel universe beckons. The average Barnes & Noble bookstore can hold about 100,000 book titles. But Amazon.com makes more money from books that are not in its top 100,000 than it does from its best-sellers. It has found a way of catering for the long tail. Each item in the tail may not sell very many, but there are so many items (typically hundreds of thousands) that their cumulative sales add up at an astonishing rate. And, of course, if this holds for physical goods like books and DVDs, it applies even more forcefully to goods that are purely digital - like the music tracks sold by Apple on its iTunes store - because with bitstreams there are effectively no physical constraints. Disk space is cheap and internet distribution costs almost nothing.
The Long Tail tries to chart the implications of the phenomenon. These are profound because they challenge the way entire industries - and the mindsets that accompany them - have been structured. Executives conditioned to trying to produce hits will have to re-tool for a world in which 'misses' are more profitable.
In the long run, the long tail should be good for everyone, because it will reverse the arteriosclerotic narrowing of choice imposed by the blockbuster culture. Retailing has always been a roller-coaster business because it involved hunting for hits. Maybe the long tail will make it more stable - and less exciting.
The Long Tail: How endless choice is creating unlimited demand; Chris Anderson; Random House; £17.99; MT price £15.99; To order, visit www.mtmagazine.co.uk
John Naughton is professor of the Public Understanding of Technology at the Open University and internet columnist of the Observer.