Posts about measurement

Good metrics, bad metrics

Screenshot 2014-03-10 at 10.28.01 AMChartbeat CEO Tony Haile writes an important piece about bad media metrics online. He pokes holes in the value of the click as the be-all-and-end-all of media measurement. He reveals that sharing turns out to be a bad measurement of engagement and value because we often don’t read what we “like” or share (we just bother other people with it). He deflates the value of native advertising, demonstrating with hard data that readers understand the difference between real content and — let’s call it what it is — advertising and they quickly abandon it.

The bottom line of Tony’s data is bad news for cynical publishers who have tried to manipulate readers with link-bait headlines and lists, and who are trying to pull the wool over advertisers’ eyes by selling them link-bait listicles and so-called native advertising. Certain emperors have no clothes. The readers know it. The advertisers will wake up and realize it.

But that’s the bad news.

Where we should turn the discussion next is to what the right metrics for media should be. As they say, you get what you measure. So what should we measure? How do we create positive feedback loops that improve the news, not degrade it as unique users, pageviews, and other relics of mass media have done?

I’ll start with the most important and most difficult thing to measure: outcomes. Were people more informed because of what we gave them? Did they accomplish what they wanted as individuals (Sally got new health insurance and saved money) or as communities (Riverdale cleaned up that messy park)? I just had breakfast with Robert Rosenthal of the Center for Investigative Reporting and he told me they start the process of reporting by considering impact and they end by trying to measure it. Why deal in bad proxies for good journalism, based on popularity, when we could get to the reason journalism should exist: to improve the world?

In his book News: A User’s Manual, Alain de Botton says that news has “the power to assemble the picture that citizens end up having of one another; the power to dictate what our idea of ‘other people’ will be like; the power to invent a nation in our imaginations.” And it has the power to help us get there. (Many more quotes in my post about the book, here.) Mark Zuckerberg says that platforms, including news, should offer communities “elegant organization.” These are higher aspirations than mere exposure.

On a tour of technology companies in Silicon Valley a few weeks ago with my dean, we talked about metrics and found different measurements being used for different platforms with different goals. Ev Williams’ Medium values total time spent reading. That is appropriate for a platform that wants to get people to explore ideas in depth — and I find I’m spending more time there reading more posts; it’s working.

Attention, in the form of time spent, is used by many in media as a measure of engagement. But that’s not always the case. Attention can also be another egocentric media metric: how many people come to look at my stuff; how many pages of my stuff do they look at; how much time do they spend with my stuff? No, sometimes, the less time spent the better. What if news were more efficient? Sometimes, spending less time to get what I want is the right metric. That metric doesn’t serve the old media business model of delivering as many eyeballs to as many ads as possible. That is why Yahoo shifted from — in the words of cofounder Jerry Yang — getting you in and out with the answer you needed as quickly as possible to instead trying to bombard you with content and keep you around as long as possible to show you as many ads as possible. Attention, in the wrong hands, can also be a corrupting metric.

Cir.ca has a fascinating metric: follows. When a reader follows a story, she is telling Cir.ca, “Please bother me and let me know when something new happens here.” That is a measure of true interest.

Similarly, Flipboard keeps track of how many people subscribe to a publication — and even to an advertiser’s publication. It also watches what people “flip” or save to read later, which strikes me as a much better indication of interest than sharing.

Google has long valued links as a digital version of citation. That has served search well. Google News also uses citations to try to infer which news organization created or is staying on top of a story — if everyone writing about Walter Reed Army Medical Center quotes the Washington Post then there’s a good chance it’s the Post’s story.

Repost.US and YouTube and now Getty Images track embeds — how many people truly want to share a video or an article because they repost it in their own space on the web. The problem with just “liking” or “sharing” on Twitter and Facebook is that there turns out to be no cost for those transactions; it’s too easy to just keep passing things on. Embedding uses my space and affected my reputation with you. I would like to see more such higher friction means of sharing that really do impute engagement.

What is engagement? It’s likely not one measure of one method of interacting with content. It could be that I spend time with something, that I interact with it or the people gathered around it (though don’t we know that comments are no indication of quality), that I save it, that I take action based on it.

We want to find good proxies for engagement in the hopes that they will lead us to indications of quality, which in turn should tell us something about the authority of the creator and the trust the public has in her. None of these is easy to measure, like “likes.”

Another word for engagement is relationship. I have been arguing that we in news should stop seeing ourselves as content factories and start seeing ourselves as members of our communities who are in the relationship business, who use what we know about people to better serve them. Thus, I ask media companies how many relationships they have with the people they serve and what they know about them — what signals they have, enabling them to improve relevance and thus value and often impact. Those are metrics that start with the public rather than with media. Those are metrics that matter.

The Nielsen revolt

I was asked by a reporter today what I thought of TV companies revolting against Nielsen and threatening to start their own measurement company. My response:

I’ve been waiting for something like this to happen as I’ve argued for sometime that the old sample-based (that is, Nielsen-family) structure of ratings simply cannot work in a niche media world. That is, there’s no way to get a large enough sample to even pretend to accurately measure audience in the unlimited number of special interests that can now be served online. Complicating this further is the on-demand nature of media now, making it hard to measure audience for things we consume via Tivo, the internet, our phones, and so on.

It’s also true that audience size matters less. The presumption of old media was that everyone in the audience saw every advertisement and that’s why ads were bought on the basis of the size of the audience. Size mattered. But today, what advertisers really want is verification that their ads reached the audience they were sold – not just in size but in relevance. (This is why Google’s model of selling clicks is so powerful; it takes the risk of matching relevant ads to audience is paid on the resultant clicks.)

Finally, the web is so much more measurable; servers know what they serve. This, too, changes the structure of measurement online.

We are seeing may industry-wide organizations falter in this new world — Nielsen, the Associated Press — because they were built to support industries that are now turned upside-down in a new media age.

Measuring influence

Steve Rubel declares dead the idea that blog links as a measure of authority. Steve likes declaring things dead; he’s kind of the grim reaper of measurements. He declared the pageview dead and now it is. This time, Steve’s point is that people do so much more than blog and so he wants to measure across those activities.

His colleague David Brain then takes up the challenge and tries to come up with a new measurement taking into account weighted activity and popularity on blogs, Twitter and company, Facebook, Del.icio.us and company, and more.

I’m all for coming up with new things to measure and new ways to measure them – things that start to capture the real value of the social-content sphere.

But I think that Steve gives an example that shows how tough it is to measure influence by the numbers. He points to Dwight Silverman of the Houston Chronicle watching a conversation on Facebook between Steve and Robert Scoble and quoting that in print, thus multiplying their influence even if the source came in a closed and ultimately, for each participant, small medium. And here’s the problem:

It’s not how many people you interact with. It’s who you interact with.

In this case, old media still has higher influence value; Rubel and Scoble influence more because they influence Silverman and Silverman watches them because they’re Rubel and Scoble. Just counting each as a person, a linker, an influencer misses the multiplying factor of standing influence and also, importantly, the qualitative impact of who’s saying what about what.

Of course, one-size-fits-all measurement is in the end meaningless. If you’re trying to influence knitters, Scobel and Rubel are not in the least influential; you want the most influential knitter, whether they click their needles in blogs or Twitter or Facebook or on video or now Pownce or whatever.

And I think there are different kinds of influence: those who start memes and those who recognize memes just as they hit critical mass and those who spread memes (not to mention those who declare them dead).

So this is more difficult to measure than merely adding up links or traffic but ultimately much richer. It has to become qualitative: ‘This person in this area had this impact on this meme (or story or discussion or idea).’

How is that done? I haven’t the faintest. I think it involves visualizing the links among people and overlaying what happens to the ideas they write about. I think we need to measure the impact people have on memes to measure the impact they have on each other. And we need to have some sense of speed and weight. It’s 3-D and, worse, needs to measure speed and time. Good luck.

: LATER: And just as I published that, I read a post by Steve Peterson of the Bivings Report about just this: trying to measure influence in a given topic about a given subject. His example is Dell Hell:

However, not all voices are equal in their importance to a company. Concerning this fact, one of issues I’ve grappled with is how to weigh general influence and influence within a specific topic.

For instance, using Dell as an example, most of the A-list bloggers very rarely discuss the company and its products directly. Granted, top blogs like Engadget and Techcrunch should interest Dell since they focus on technology, but what about other blogs like Boing Boing and The Huffington Post?

I use Dell as an example since the company has had to deal with a top blog that doesn’t focus on its arena. Remember Dell Hell?

Although Jeff Jarvis and his BuzzMachine blog are prominent, they focus on media, not technology. Thus, they typically shouldn’t worry Dell, but when Jarvis blogged about his “Dell Hell,” the rules changed. In fact, sometimes when bloggers (especially an A-lister) complain about a company and its products, word can spread fast. Sometimes even the mainstream media picks up on such rants.

Measurement is tough in situations like Dell Hell. Does Dell need to devote resources to scrupulously follow BuzzMachine? No, since Jarvis mainly blogs about media and not computer hardware and software. However, Jarvis was worth Dell’s attention for a while.

This also says that spheres of influence are far from fixed. With the cumulative speed of links, one can spread or influence and idea to new people and people can coalesce around that idea (more than that person).

How can a company determine which bloggers who don’t focus on the company and its field require their attention? Then, when should they start and stop monitoring such blogs?

The problem with measurement

Measurement is holding back so much progress online — both because there’s not enough of it and because what there is of it is too often tracking the wrong things. Without good measurement, we won’t get money and without money we won’t be able to sustain so much of what we want to do online.

First, about measuring the wrong things. Nielsen just switched from measuring page views to time and that’s a step in the right direction — about a year and a half too late. It’s finally acknowledged that the pageview is dead thanks to dynamic content that makes act like shows. Heather Green talks about how time is also a meaningless measure, though, because we simply spend more time with some applications (e.g., IM) than others (e.g., content). Time is not a good proxy for engagement. This gets much more complicated when content and functionality becomes widgetized and distributed. There is no means of measuring traffic/attention/engagement for widgets. And this also means that the site itself becomes obsolete as content and functionality are distributed anywhere and everywhere; I predict that we’ll soon see brands that essentially don’t have sites but become huge because they are distributed (think apps on Facebook that have no home page). I’ve often said that Google’s audience is many times what is reported because Google distributes itself as widgets — ads, maps, feeds. Google doesn’t care because it’s measuring only one thing: revenue. And Google knows everything it needs to know. We don’t.

Now about what’s not being measured. There’s a very long list of applications — RSS, widgets, mobile, apps — and kinds of content — video, podcasts — but also of new sorts of measurements — such as influence, meme-starting, involvement, creation, engagement, popularity — that aren’t even being tackled. And there are new dimensions that need to be explored, such as measuring a person’s trust, influence, or even fame across many platforms, sites, applications, and so on.

If it’s not measured, advertisers won’t buy it. And if advertisers won’t buy it, companies won’t build it. And that’s the problem.

Well, but that’s only somewhat true. Big, old companies won’t build it. New inventors will and that gives them the headstart they need. YouTube, Facebook, Twitter, Pownce, and so many innovations started without measurement — or evenue. But they had to have the courage of their vision and enough money to make it big first. What else has died just because conservative advertisers and agencies and their outmoded measurement companies — who are still using sample panels that simply will not work in a nichefied world — hadn’t caught up with them?

I’ve suggested before the need for a measurement summit. Maybe that’s one way to tackle this. Or perhaps we need to find ways to better educate advertisers (but we’ll never educate them out of wanting metrics). Or perhaps this is something academics should take on with foundation help because it will support invention: measure it and they will come.

: LATER: More from Heather.

Friends’ friends

Market IQ says it has a new method for measuring influence that does a degree of separation beyond Technorati influence. They say it matters how many links the people linking to you have.

The need for a measurement summit

Comscore and Federated Media (which sells some ads on this blog) have teamed up to try to improve measurement in the long tail of social and niche media online. And that’s good.

Except I argue that the panel means of measurement is doomed to miserable failure in the mass of niches. You cannot possible build a panel large and varied enough to get reliable measurement of the audience and traffic of millions — even thousands — of sites, especially when we get the means to tie together lots of those small sites into networks.

What I hope they do is honestly and harshly look at their stats from their panel versus the server stats of the sites — especially the smaller sites, not the much-easier-to-measure big boys like Digg and BoingBoing — and realize that the panel just doesn’t work.

What we need, I’ve long argued, is standard metrics reported from the sites’ servers or from snoopers on pages and verified by a service such as Comscore or Nielsen. Old methods will not work in this new world. The same goes for Nielsen, which is buying the rest of Netratings.

And whilel we’re at it, let’s figure out the new measurements that capture the unique value of this new medium: authority, speed, connectedness… The page view is dead.

I think it’s time for a measurement summit: Bring together the measurement companies, the advertisers and their agencies (buyers), the sites’ reps (sellers), the media sites, and technology companies and let’s hammer out some standards and methods for measurement. This will only work if we have open standards with analytics (like Comscore and Nielsen) building value atop that common data. Otherwise, we end up in a world that will continue to confuse and scare advertisers — and their money — away.

Guardian column: Death of the page view

My Guardian column this week is a sanded and polished version of recent posting and linking about the death of the page view and, with it, the mass. (Nonregistration version here.)

Just when we were getting used to it, the page view has been declared dead. There are many reasons for its passing, having to do with how web pages are now made and how web content is now distributed. But there is one seismic implication to this – in media, mass is over. Size doesn’t matter.

It was only about a decade ago that I sat on a dreadfully boring committee of the American Audit Bureau of Circulations debating how to define a page view (rather than a hit) as the elemental measurement of new media. This body, which blesses the circulation counts of print products, tried to replicate its world-view online, verifying the circulation – that is, the audience and traffic – of internet properties. But as it turned out, no advertiser or publisher wanted these audits. All marketers cared about was verifying whether they got what they paid for: views, people, clicks. You see, overall circulation mattered only when you and your ads were stuck in the same pages with many other advertisers and you all got the same audience, whether that audience gave a damn about you or not. But now, online, you could find better ways to reach just the people you wanted or who wanted you. Thus, travel advertisers needn’t care about the circulation of Guardian Unlimited, only about who saw their ads on travel pages.

In recent times, the situation has grown more complicated because, on the web, a page is no longer a page. Video can be served on a page, but it is measured in time, not space. Flash and Ajax technology can make any individual page many levels deep, allowing users to interact with content – navigating maps, ordering merchandise, viewing slideshows, chatting – without ever leaving the page. So the activities that once would have added up to a dozen page views will now count as only one. This is having a significant impact on businesses such as Yahoo, which are using these technologies to improve the user experience, reducing clicks in the hopes of increasing time on the site or satisfaction or loyalty. But this reduces page views and with that, bragging rights and, in some cases, revenue.

Now add to this the widgetisation of the web. Content may be displayed not only on your pages but also in widgets – boxes, gadgets and applications – that are embedded in pages elsewhere. This is how much of MySpace is built and how YouTube spreads video all over the internet. The audience becomes the distributor. How do you count that?

And consider Google AdSense modules that are spread all across the web, from NYTimes.com to my humble blog. Shouldn’t each of those be counted as Google page views since Google revenue is attached? Doesn’t that make Google look even more gigantic than it already is? What this really means is that in the new distributed media economy, owning a site doesn’t matter so much as enabling a network. This, in a nutshell, is why Yahoo, the centralised media property, is at a disadvantage versus Google, the distributed network.

Things get even crazier when you consider that if you make a good commercial, the public will distribute it for you on YouTube – advertising becomes content. Now that is really an upside-down world.

Finally, consider the impossibility of the old means of measurement. TV ratings were based on a sampling that determined the proportion of the audience watching, say, channel 2 or 4. But in this new niche world, no sample can possibly be large enough to measure millions of blogs or online TV shows.

But more fundamental than all this is, again, that size doesn’t matter anymore, not in media. Oh, yes, the movie with the biggest box office or the book with the biggest sales still makes the most money – for now at least. But in more and more of the media, mass measurements are obsolete because we are now fragmented into the mass of niches. And the truth is that we, the audience, never cared how many more people were watching what we watched. And advertisers don’t care so much what we’re watching so long as we’re watching them.

In a world of so many choices, the audience care about trust, taste, relevance, usefulness, not ratings. And advertisers care more about targeting, efficiency, engagement, branding and return on investment. These are better measurements than print circulation or broadcast ratings or online page views. And so now, publishers, advertisers and technologists must catch up and change their yardsticks for success yet again. It is time to measure quality over quantity.

Size doesn’t matter: The distributed media economy

No, size does not matter, not in media, not anymore.

I know that’s counterintuitive and counter everything we’ve assumed about mass media. But today what matters is reaching the right people by the right means. That has always been the case. Only now, thanks to connected, collaborative media, it’s finally possible.

I’ll pull together a lot of links around this topic below. But most of them are still trying to measure mass: the new pageview, the
new audience count, the new click. I say the change we’re facing is much bigger than just the obsolescence of the pageview, much more fundamental: Size doesn’t matter. Relevance, credibility, and attraction do.

Instead of measuring quantity, we have to measure quality. And only when we do that will the true value of these new media be unlocked for everyone.

Some of the discussion that is boiling up out there:

* The end of the pageview: Steve Rubel has been doing a good job hammering on the anti-meme that the pageview is over: “The page view does not offer a suitable way to measure the next generation of web sites. These sites will be built with Ajax, Flash and other interactive technologies that allow the user to conduct affairs all within a single web page – like Gmail or the Google Reader. This eliminates the need to click from one page to another.” See, for example, Yahoo grappling with the impact of the unpage. This is not entirely new; it was a problem I grappled with on refreshing chat pages a decade ago. But the phenomenon is growing in both an Ajaxed web and a Flashed video world: What’s a page now? What’s a view? What’s a viewer? A decade ago, I spent months on a tortuous committee of the Audit Bureau of Circulations answering just those questions. Now, it doesn’t matter, or at least, it matters less and less.

* Targeting and verification matter more than size: When I sat in those endless ABC committee meetings, our aim was to come up with the standards against which to audit the circulation or audience — old terms — of web sites. But that effort was eventually abandoned because advertisers didn’t care about verifying the size of a site; only publishers cared about those bragging rights and not enough to pay for auditing them. As it turned out, advertisers cared only about auditing their own flights of ads: ‘Did I get what I paid for (whether that was people or views or clicks or actions or demos or branding)?’ You see, circulation mattered only when you were stuck in the same pages as all the advertisers and you all got the same audience whether that audience gave a damn about you or not. But online, you could find ever-better ways to reach just the people you wanted or who wanted you. Travel advertisers didn’t need to care about the circulation of NewYorkTimes.com, only about who saw its ads in the travel section. Oh, yes, advertisers are still buying the old way, but that’s because it’s more convenient — albeit far less efficient — to buy us in bulk. But the mass is gone. Size doesn’t matter.

* The widgetization of the web: Niall Kennedy called it back in July: Pages are now made up of widgets that operate like multiple pages themselves. But this is about more than adding cool stuff to your site. See how MySpace is built with widgets from elsewhere and how Flickr is spread via widgets. This is a new means of distribution.

* The people are your distributors: One day this week, seven of the top 10 viral videos — determined by links and embeds, that is, by recommendations rather than just traffic — were performances by James Brown, following his death. (By the way, I didn’t see these videos on the most-viewed lists on YouTube; those are the old, mass lists people still look at but they’re pretty much meaningless). This tells me that the people will distribute your stuff if given a chance; they jumped onto word of Brown’s death and they served relevance. Note well that you don’t need everyone doing this; even as Wikipedia’s content is made by the fabled 1 percent of its users, so can the new networks of information be driven by 1 percent of their members.

* The distributed media economy is taking over: This last week, Google was said to surpass Yahoo as the second most visited site on the internet after Microsoft; this comes after MySpace surpassed Yahoo in pageviews. But the truth is that Google surpassed them all long ago, for Google is not a site; it is a platform. Every piece of Google — like the ads on this page — counts as another pageview, if you’re still counting them. This is why I keep saying that Yahoo is the last old-media company, relying on controlling content and marketing to attract an audience to see ads, but Google is the first distributed platform, no longer making the people come to it but going to the people wherever they are. And see James Brown: The people will take you to the people, if you’re good and if you’re lucky and if you let them.

Add to this the notion that advertising can be content (the viral ad) and that ad creative gets tangled up again with media and distribution (being on MySpace is itself the brand statement) and you continue to unravel all the old assumptions about the media economy; see Scott Karp arguing that the page view will be dethroned by innovation in advertising.

And mess things up even more when you start tearing apart the methodologies of measurement. Fred Wilson, on the board of Comscore, begins to address this. I argue that any paneled, sampled measurement scheme will simply not work in this new world. Full stop. The Nielsen method of putting together a sample of people who represented the rest of us cannot work in the mass of niches, for you cannot have a sample that is ever large enough to measure the tail. You can’t measure quantity.

So pageviews are obsolete already, thanks to Ajax and other unpage technologies and to the widgetization of content, funtionality, and branding: Again, what’s a ‘page’? Audience measurements are obsolete, at last, thanks to the fact that the former consumer is now also the creator and distributor: What’s an ‘audience’? Mass measurements are dead, thank God, because we are now joyfully fragmented into the mass of niches: Who’s a ‘user’?

The truth is that we, the former audience, have long paid only scant attention to the old, quantitative measurements: Box office numbers and Nielsen ratings were curiosities. We have always measured, instead, relevance, trust, usefulness, interest, attraction, action, value. Those are the measurements that matter, always have been, only now media must catch up to us. And when media and marketers do, they will give us greater value and get more in return.

This is about far more than the damned pageview.

See also:
* Scott Karp on the fallout for media and creative ad agencies.
* Michael Parekh.
* Evan Williams last summer on the dying pageview.