Big Data and the Promise of the Long Tail

Tom Breur

5 March 2017

The introduction of internet promised to change everything. From 1995 to 2001 stock prices for internet startups went through the roof, and as everybody was shaking their heads, the next quarter market valuations for companies that had yet to go live, went up even further. Of course that couldn’t last. After the bubble burst in 2001, there was a shakedown, some interesting penny stocks, but more importantly: a rethinking of why the internet was supposed to turn capitalism on its head.

Having “one” internet wasn’t enough, so along came Web 2.0, and later futurists like Tim Berners Lee coined Web 3.0, more commonly referred to as the semantic web. As skeptical or cynical as I may sound about some of these hype cycles, I genuinely believe that the internet has revolutionized distribution models. I also believe it will continue to drive further innovation. This can become manifest in something mundane like shopping for a pair of used skates on Craigslist, or more advanced solutions like crowdsourcing, or price comparison for critical (high ticket) purchases. In the long run, I am convinced this will evolve to true multi-channel delivery of many services and goods.

Advances in analytics have dramatically raised the bar when it comes to personalization. As consumers get used to this higher standard, naturally their expectations are raised. An area where personalization has transformed to the next level is “long tail selling”, as laid out in Chris Anderson’s book (2006) “The Long Tail: Why the Future of Business is Selling More of Less”. And this isn’t just about selling only, it is about finding niches for products that may have takers, but that are hard to find. In the old days, Top 40 radio meant playing the same hits over and over again. With the advent of streaming services like Pandora, Spotify, Deezer, Tidal, and several others, a whole new market for (much) less popular songs has opened up. Note that the promise of opening up markets for “odd” songs rests on the premise of being able to find their respective niche takers!

As a distribution channel that can reach far and wide, the internet is second to none. When you listen to music on Spotify, and want “skip” a track, you essentially provide the system real-time feedback about your preference (or dislike, as in this case). With over 100 million active users (40 million paying subscribers), Spotify has an extremely powerful base to gather valid data (pertaining to actual usage), and to try out alternative algorithms to select and test new music tracks to see how people respond – and massively parallel! Users like me are often genuinely surprised when they hear something familiar that hadn’t been played through Spotify, yet. It’s as if the system actually knows you. Analytics’ finest hour.

By reaching out to the far corners of commercial markets, referred to as niches, analytics helps to actually grow the effective market. There is only so much demand for the mainstream blockbusters, Top 40 hits, etc. By finding a reasonable fit for the songs that have extremely low a priori chance of being liked (unless you find the right maniac for them), you leverage both the internet as a remarkable distribution channel, in conjunction with the power of analytics. So both your playlist gets expanded, as well as giving the consumer a remarkable sense of feeling understood.

When it comes to serving up the right product, to the right customer, at the right time, both scale and intelligence matter. You need “scale” because unless you have a gargantuan database, you won’t have access to (enough) niche customers. But you also need “intelligence” to accurately find that perfect rare match. Data science has come a long way. Consumers’ (much) higher expectations are the living testimony of our success. The market, ever more compelling user experiences, taught customers to expect more. As we venture into new and innovative applications in years to come, I think we have only scratched the surface.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s