Posts about measurement

Good metrics, bad metrics

March 10, 2014 by Jeff Jarvis
, , , , , , ,

Screenshot 2014-03-10 at 10.28.01 AMChartbeat CEO Tony Haile writes an important piece about bad media metrics online. He pokes holes in the value of the click as the be-all-and-end-all of media measurement. He reveals that sharing turns out to be a bad measurement of engagement and value because we often don’t read what we “like” or share (we just bother other people with it). He deflates the value of native advertising, demonstrating with hard data that readers understand the difference between real content and — let’s call it what it is — advertising and they quickly abandon it.

The bottom line of Tony’s data is bad news for cynical publishers who have tried to manipulate readers with link-bait headlines and lists, and who are trying to pull the wool over advertisers’ eyes by selling them link-bait listicles and so-called native advertising. Certain emperors have no clothes. The readers know it. The advertisers will wake up and realize it.

But that’s the bad news.

Where we should turn the discussion next is to what the right metrics for media should be. As they say, you get what you measure. So what should we measure? How do we create positive feedback loops that improve the news, not degrade it as unique users, pageviews, and other relics of mass media have done?

I’ll start with the most important and most difficult thing to measure: outcomes. Were people more informed because of what we gave them? Did they accomplish what they wanted as individuals (Sally got new health insurance and saved money) or as communities (Riverdale cleaned up that messy park)? I just had breakfast with Robert Rosenthal of the Center for Investigative Reporting and he told me they start the process of reporting by considering impact and they end by trying to measure it. Why deal in bad proxies for good journalism, based on popularity, when we could get to the reason journalism should exist: to improve the world?

In his book News: A User’s Manual, Alain de Botton says that news has “the power to assemble the picture that citizens end up having of one another; the power to dictate what our idea of ‘other people’ will be like; the power to invent a nation in our imaginations.” And it has the power to help us get there. (Many more quotes in my post about the book, here.) Mark Zuckerberg says that platforms, including news, should offer communities “elegant organization.” These are higher aspirations than mere exposure.

On a tour of technology companies in Silicon Valley a few weeks ago with my dean, we talked about metrics and found different measurements being used for different platforms with different goals. Ev Williams’ Medium values total time spent reading. That is appropriate for a platform that wants to get people to explore ideas in depth — and I find I’m spending more time there reading more posts; it’s working.

Attention, in the form of time spent, is used by many in media as a measure of engagement. But that’s not always the case. Attention can also be another egocentric media metric: how many people come to look at my stuff; how many pages of my stuff do they look at; how much time do they spend with my stuff? No, sometimes, the less time spent the better. What if news were more efficient? Sometimes, spending less time to get what I want is the right metric. That metric doesn’t serve the old media business model of delivering as many eyeballs to as many ads as possible. That is why Yahoo shifted from — in the words of cofounder Jerry Yang — getting you in and out with the answer you needed as quickly as possible to instead trying to bombard you with content and keep you around as long as possible to show you as many ads as possible. Attention, in the wrong hands, can also be a corrupting metric.

Cir.ca has a fascinating metric: follows. When a reader follows a story, she is telling Cir.ca, “Please bother me and let me know when something new happens here.” That is a measure of true interest.

Similarly, Flipboard keeps track of how many people subscribe to a publication — and even to an advertiser’s publication. It also watches what people “flip” or save to read later, which strikes me as a much better indication of interest than sharing.

Google has long valued links as a digital version of citation. That has served search well. Google News also uses citations to try to infer which news organization created or is staying on top of a story — if everyone writing about Walter Reed Army Medical Center quotes the Washington Post then there’s a good chance it’s the Post’s story.

Repost.US and YouTube and now Getty Images track embeds — how many people truly want to share a video or an article because they repost it in their own space on the web. The problem with just “liking” or “sharing” on Twitter and Facebook is that there turns out to be no cost for those transactions; it’s too easy to just keep passing things on. Embedding uses my space and affected my reputation with you. I would like to see more such higher friction means of sharing that really do impute engagement.

What is engagement? It’s likely not one measure of one method of interacting with content. It could be that I spend time with something, that I interact with it or the people gathered around it (though don’t we know that comments are no indication of quality), that I save it, that I take action based on it.

We want to find good proxies for engagement in the hopes that they will lead us to indications of quality, which in turn should tell us something about the authority of the creator and the trust the public has in her. None of these is easy to measure, like “likes.”

Another word for engagement is relationship. I have been arguing that we in news should stop seeing ourselves as content factories and start seeing ourselves as members of our communities who are in the relationship business, who use what we know about people to better serve them. Thus, I ask media companies how many relationships they have with the people they serve and what they know about them — what signals they have, enabling them to improve relevance and thus value and often impact. Those are metrics that start with the public rather than with media. Those are metrics that matter.

The Nielsen revolt

August 14, 2009 by Jeff Jarvis
, , ,

I was asked by a reporter today what I thought of TV companies revolting against Nielsen and threatening to start their own measurement company. My response:

I’ve been waiting for something like this to happen as I’ve argued for sometime that the old sample-based (that is, Nielsen-family) structure of ratings simply cannot work in a niche media world. That is, there’s no way to get a large enough sample to even pretend to accurately measure audience in the unlimited number of special interests that can now be served online. Complicating this further is the on-demand nature of media now, making it hard to measure audience for things we consume via Tivo, the internet, our phones, and so on.

It’s also true that audience size matters less. The presumption of old media was that everyone in the audience saw every advertisement and that’s why ads were bought on the basis of the size of the audience. Size mattered. But today, what advertisers really want is verification that their ads reached the audience they were sold – not just in size but in relevance. (This is why Google’s model of selling clicks is so powerful; it takes the risk of matching relevant ads to audience is paid on the resultant clicks.)

Finally, the web is so much more measurable; servers know what they serve. This, too, changes the structure of measurement online.

We are seeing may industry-wide organizations falter in this new world — Nielsen, the Associated Press — because they were built to support industries that are now turned upside-down in a new media age.

Measuring influence

July 17, 2007 by Jeff Jarvis

Steve Rubel declares dead the idea that blog links as a measure of authority. Steve likes declaring things dead; he’s kind of the grim reaper of measurements. He declared the pageview dead and now it is. This time, Steve’s point is that people do so much more than blog and so he wants to measure across those activities.

His colleague David Brain then takes up the challenge and tries to come up with a new measurement taking into account weighted activity and popularity on blogs, Twitter and company, Facebook, Del.icio.us and company, and more.

I’m all for coming up with new things to measure and new ways to measure them – things that start to capture the real value of the social-content sphere.

But I think that Steve gives an example that shows how tough it is to measure influence by the numbers. He points to Dwight Silverman of the Houston Chronicle watching a conversation on Facebook between Steve and Robert Scoble and quoting that in print, thus multiplying their influence even if the source came in a closed and ultimately, for each participant, small medium. And here’s the problem:

It’s not how many people you interact with. It’s who you interact with.

In this case, old media still has higher influence value; Rubel and Scoble influence more because they influence Silverman and Silverman watches them because they’re Rubel and Scoble. Just counting each as a person, a linker, an influencer misses the multiplying factor of standing influence and also, importantly, the qualitative impact of who’s saying what about what.

Of course, one-size-fits-all measurement is in the end meaningless. If you’re trying to influence knitters, Scobel and Rubel are not in the least influential; you want the most influential knitter, whether they click their needles in blogs or Twitter or Facebook or on video or now Pownce or whatever.

And I think there are different kinds of influence: those who start memes and those who recognize memes just as they hit critical mass and those who spread memes (not to mention those who declare them dead).

So this is more difficult to measure than merely adding up links or traffic but ultimately much richer. It has to become qualitative: ‘This person in this area had this impact on this meme (or story or discussion or idea).’

How is that done? I haven’t the faintest. I think it involves visualizing the links among people and overlaying what happens to the ideas they write about. I think we need to measure the impact people have on memes to measure the impact they have on each other. And we need to have some sense of speed and weight. It’s 3-D and, worse, needs to measure speed and time. Good luck.

: LATER: And just as I published that, I read a post by Steve Peterson of the Bivings Report about just this: trying to measure influence in a given topic about a given subject. His example is Dell Hell:

However, not all voices are equal in their importance to a company. Concerning this fact, one of issues I’ve grappled with is how to weigh general influence and influence within a specific topic.

For instance, using Dell as an example, most of the A-list bloggers very rarely discuss the company and its products directly. Granted, top blogs like Engadget and Techcrunch should interest Dell since they focus on technology, but what about other blogs like Boing Boing and The Huffington Post?

I use Dell as an example since the company has had to deal with a top blog that doesn’t focus on its arena. Remember Dell Hell?

Although Jeff Jarvis and his BuzzMachine blog are prominent, they focus on media, not technology. Thus, they typically shouldn’t worry Dell, but when Jarvis blogged about his “Dell Hell,” the rules changed. In fact, sometimes when bloggers (especially an A-lister) complain about a company and its products, word can spread fast. Sometimes even the mainstream media picks up on such rants.

Measurement is tough in situations like Dell Hell. Does Dell need to devote resources to scrupulously follow BuzzMachine? No, since Jarvis mainly blogs about media and not computer hardware and software. However, Jarvis was worth Dell’s attention for a while.

This also says that spheres of influence are far from fixed. With the cumulative speed of links, one can spread or influence and idea to new people and people can coalesce around that idea (more than that person).

How can a company determine which bloggers who don’t focus on the company and its field require their attention? Then, when should they start and stop monitoring such blogs?

The problem with measurement

July 11, 2007 by Jeff Jarvis

Measurement is holding back so much progress online — both because there’s not enough of it and because what there is of it is too often tracking the wrong things. Without good measurement, we won’t get money and without money we won’t be able to sustain so much of what we want to do online.

First, about measuring the wrong things. Nielsen just switched from measuring page views to time and that’s a step in the right direction — about a year and a half too late. It’s finally acknowledged that the pageview is dead thanks to dynamic content that makes act like shows. Heather Green talks about how time is also a meaningless measure, though, because we simply spend more time with some applications (e.g., IM) than others (e.g., content). Time is not a good proxy for engagement. This gets much more complicated when content and functionality becomes widgetized and distributed. There is no means of measuring traffic/attention/engagement for widgets. And this also means that the site itself becomes obsolete as content and functionality are distributed anywhere and everywhere; I predict that we’ll soon see brands that essentially don’t have sites but become huge because they are distributed (think apps on Facebook that have no home page). I’ve often said that Google’s audience is many times what is reported because Google distributes itself as widgets — ads, maps, feeds. Google doesn’t care because it’s measuring only one thing: revenue. And Google knows everything it needs to know. We don’t.

Now about what’s not being measured. There’s a very long list of applications — RSS, widgets, mobile, apps — and kinds of content — video, podcasts — but also of new sorts of measurements — such as influence, meme-starting, involvement, creation, engagement, popularity — that aren’t even being tackled. And there are new dimensions that need to be explored, such as measuring a person’s trust, influence, or even fame across many platforms, sites, applications, and so on.

If it’s not measured, advertisers won’t buy it. And if advertisers won’t buy it, companies won’t build it. And that’s the problem.

Well, but that’s only somewhat true. Big, old companies won’t build it. New inventors will and that gives them the headstart they need. YouTube, Facebook, Twitter, Pownce, and so many innovations started without measurement — or evenue. But they had to have the courage of their vision and enough money to make it big first. What else has died just because conservative advertisers and agencies and their outmoded measurement companies — who are still using sample panels that simply will not work in a nichefied world — hadn’t caught up with them?

I’ve suggested before the need for a measurement summit. Maybe that’s one way to tackle this. Or perhaps we need to find ways to better educate advertisers (but we’ll never educate them out of wanting metrics). Or perhaps this is something academics should take on with foundation help because it will support invention: measure it and they will come.

: LATER: More from Heather.

Friends’ friends

February 14, 2007 by Jeff Jarvis

Market IQ says it has a new method for measuring influence that does a degree of separation beyond Technorati influence. They say it matters how many links the people linking to you have.