Recommending the tail

Wharton and O’Reilly just released two provocative reports on whether social distribution and recommendation really get into the long tail.

First, O’Reilly’s on the distribution of Facebook apps:

The good news has already been widely disseminated: there are nearly 5000 Facebook applications, and the top applications have tens of millions of installs and millions of active users. The bad news, alas, is in our report: 87% of the usage goes to only 84 applications! Only 45 applications have more than 100,000 active users. This is a long tail marketplace with a vengeance — but unfortunately, the economic models (for developers at least, though not for Facebook itself) all rely on getting into the very short head.

I think there are a few reasons for that. First, the Facebook platform is so damned new. If the same analysis of the entire web had been made in December, 1994, two months after Netscape’s release, it would have shown that Netscape got most of the attention along with a camera on a coffee pot. It took a long time for the Web to develop its incredible depth: its tail. The Facebook platform is very much in its infancy. It’s far too soon to draw any grand conclusions.

More substantively, I think one reason for this undistributed distribution is the nature of social apps: They gain in value the more that people — especially you know — use them, and so the community is uniquely motivated to create blockbusters. It’s one matter to simply recommend things to people (more on that from Wharton in a minute); it doesn’t really affect you if more people watch the movie you recommend, except that you feel as if you’re part of a trend and maybe you can discuss it with them. Those are light motives. By contrast, many Facebook apps are all but useless if your friends don’t use them; that’s the social in it. This creates more of a gathering point than mere recommendation.

I think there’s a lesson in this for old, blockbuster-oriented economies — entertainment and media, mainly: How do you improve your product for all by having more people involved in it? And how does that motivate people to spread it for you? We have seen this happening in online forums: the more people who are involved, the more people get involved (though there is a tipping point; you can have too many people). I wonder whether collaborative media could take on this effect. Lonely Girl 15 may be an example: people made media around the media and spread the original along with their creations. How can newspapers and TV shows do likewise? How does the collaboration and the involvement of your friends improve the product and how then do you get your friends involved? If I were trying to produce a social news or entertainment product, I’d investigate that formula.

Now shift to mere recommendation. The Wharton report (via PaidContent) says that as presently implemented, automated recommendation systems tend to cluster people around products and create blockbusters.

Online retailers may be shooting themselves in the tail — the long tail, that is, according to Kartik Hosanagar, Wharton professor of operations and information management, and Dan Fleder, a Wharton doctoral candidate, in new research on the “recommenders” that many of these retailers use on their websites. Recommenders — perhaps the best known is Amazon’s — tend to drive consumers to concentrate their purchases among popular items rather than allow them to explore and buy whatever piques their curiosity, the two scholars suggest in a working paper titled, “Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity.”

Hosanagar and Fleder argue that online recommenders “reinforce the blockbuster nature of media.” And they warn that, by deploying standard designs, online retailers may be recreating the very phenomenon — circumscribed media purchasing choices — that some of them have bragged about helping consumers escape.

The problem is with automated recommendations and that a critical point:

“Because common recommenders recommend products based on sales and [consumer] ratings, they cannot recommend products with limited historical data, even if they would be rated favorably,” they write. “This can create rich-get-richer effects for popular products and vice-versa for unpopular ones, which results in less diversity.”

That could be solved or balanced, I think, if you shift to reliance on human recommendations: ‘My friend Fred finds good stuff for me…. My friend Sally finds better stuff than Fred…. My friend Jeff has no taste.’ Then a critical mass of historical data doesn’t really matter; relationships and taste and shared knowledge do. And we find the friends who like the stuff we like. We live in the tail. We can also live in the head of the curve: We all watch American Idol, too. More on this later…

  • http://blog.ipdev.net Jeremy Penston

    Hi,

    I looked at this on my blog a few months back looking specifically at Online Video. I compared YouTube with DVD rentals and could see that YouTube viewing was actually much more concentrated than was the case with DVD rentals.

    I only had access to the Top 20 YouTube data, but the 20th most popular YouTube video (all time, at the time of writing) had 18% of the showings of the 1st most popular.

    On the DVD rental side, the 20th largest grossing rental (in 2006, 1 year approximately equivalent to YouTube’s all time at the time of writing) generated 79% of the amount paid for the 1st most popular.

    I know comparing viewing and gross revenue is inexact, but will do for this purpose. The conclusion was that recommendations make the hot hotter and do nothing for the long tail.

    Think of it as a long tail on an elephant… Long tail, but massive body :-)

    Jeremy

  • http://makethelogobigger.blogspot.com bg

    Interesting. I think Print/TV trying to replicate the lonelygirl15 model though is an apples and oranges comparison some ways. Her content originated online and subsequently spread there. Print/TV have their content show up subsequently online. Sure, some TV features hiughlights of other shows, but the intenet changes the equation and gives them another outlet. The interent doesn’t need print and TV for all of it’s content to survive. It helps, and more and more TV content has shown up there, but it’s not critical to the survivalof the net.

    TV even has to promote the internet to compete. In the past, would print ever have supported another media like that? TV reruns content online. Fans repurpose it for mashups on YouTube.

  • http://makethelogobigger.blogspot.com bg

    (How’s my spelling? Yikes. Call 1-800-TYPO.)

    ;-p

  • http://metzmash.com Adam Metz

    Jeff, you’ve got a great point about wanting to socialize the reccomendation data, but there’s one critical flaw. Unless Amazon, or other web retail sites that have a high “reccomender index,” aligns with OpenID or the social network that leverages individuals against the social graph (in my case, Facebook), then they’re stuck in the crummy “rich get richer” model that you desribe.

    The reccomendation data is only as good as the social graph it’s leveraged against. So, will FB become the new Amazon?

  • http://www.choicestream.com Kyle Austin

    Hey Jeff,

    Glad you took a look at recommendation engines. However, it’s unfortunate that the authors of the Wharton report don’t look at next generation recommendation engines that actually assist in helping people dive into the long tail.

    I work with the Racepoint Group and represent a company called ChoiceStream in Cambridge, MA that is working on some of this new recommendation technology.

    ChoiceStream’s scientists, statisticians and personalization experts created a new, patent-pending approach to personalization called Attributized Bayesian Choice Modeling (ABCM).

    ChoiceStream’s ABCM is based on the principle that in order to provide truly accurate, useful recommendations, a personalization system must understand not just what users like, but why they like it. By using techniques to classify content and products in terms of attributes people care about—attempting to represent content using the same characteristics that consumers consider when evaluating it—ChoiceStream’s ABCM-based solution matches each individual’s needs and interests with the content they are most likely to enjoy.

    ChoiceStream currently powers recommendations for Blockbuster, Overstock, DirecTV, Comcast and Yahoo! – among others.

    Blockbuster customer Robb Hecht’s experience with being recommended and enjoying a movie on the long tail was documented in this piece (http://online.wsj.com/article/SB118584111148282848.html) in the Wall Street Journal in July.

    I’d love to connect you with the folks from ChoiceStream to discuss the idea of intelligently recommending relevant content from the long tail as well as how they are eyeing making recommendations a social experience online and on future set-top boxes.

    Best,
    Kyle