Posts about chartbeat

Geeks Bearing Gifts: Metrics

The New York Times’ investigation into the dangers and abuses nail parlor workers are subjected to just led to Gov. Andrew Cuomo ordering inspections and, where necessary, closure of offending salons. That is media impact: improving people’s lives. Media impact is not what we measure now: pageviews, unique users, likes, shares, and all that. In the latest (and next-to-last) free chapter of Geeks Bearing Gifts, I offer my view of the metrics we should be using. Here’s a snippet; read the rest here. (And here’s a related post I wrote after the book about internally vs. externally focused journalism.)

Screenshot 2015-05-11 at 8.51.23 AM

There is a richer set of metrics that matter to the mission of journalism — metrics around impact and accomplishment. These metrics must start not with us but with the people we serve. They must measure whether they meet their needs and accomplish their goals. Thus journalism built around impact doesn’t start with producing media’s content; it starts with listening to the public’s wants, needs, and goals so we can measure our success ultimately against whether those goals are achieved. (Therein lies the heart of our rationale for starting a new degree at CUNY in social journalism, a journalism built first on listening to and understanding and then serving a community; more on that in the afterword.)

Impact is terribly difficult to measure. It cannot easily be reduced to a number in a chart. But that is precisely why we should pursue these measures, because they force us to see people as individuals again; they force us to listen. Columbia University scholar James Carey argued that media were corrupted by the desire to reduce people to mere numbers — pro vs. anti, red vs. blue — as exemplified by the public-opinion poll. Polling, he said, “was an attempt to simulate public opinion in order to prevent an authentic public opinion from forming.” His alternative: conversation with the public. “I believe we must begin with the primacy of conversation,” Carey wrote. “It implies social arrangements less hierarchical and more egalitarian than its alternatives.” Yes, we have conversations online now, plenty of conversation in bottomless comments sections under our articles. But those comments are still mediacentric, reacting to the content we produce. No, the proper conversation journalists should have must start with the public. These conversations must start not with us speaking but with us listening. Thus I come to better understand my friend Jay Rosen’s interpretation of Carey’s admonitions to journalists: “The press does not ‘inform’ the public,” Rosen said. “It is ‘the public’ that ought to inform the press. The true subject matter of journalism is the conversation the public is having with itself.”

Now I return to amend the relationship metrics I proposed above: How many people have we met — where possible, face-to-face? How rich and informative were our conversations with them? How well do we understand them and their communities? How well have we heard or discerned their needs and goals? That is the starting point.

Only then, only when we have listened well, can we ask the next questions: What does an individual or a community need to accomplish its goals? How can we help? Is it information you need? How can we help you share what you already know? If what you need is not known, then how can we bring reporting to the task? Is it understanding you need? Then how can we bring explanation and context or education? Is it functionality you need? Do we have the skills to implement or build that? Is it organization you need? Can we help convene a community to deliberate and accomplish its goals?

Only after we have contributed to a community’s efforts to reach its own goals can we begin to think of measuring success — that is, impact….

Read the rest here.

If you can’t wait for the rest of the book, then you can buy it here.

Good metrics, bad metrics

Screenshot 2014-03-10 at 10.28.01 AMChartbeat CEO Tony Haile writes an important piece about bad media metrics online. He pokes holes in the value of the click as the be-all-and-end-all of media measurement. He reveals that sharing turns out to be a bad measurement of engagement and value because we often don’t read what we “like” or share (we just bother other people with it). He deflates the value of native advertising, demonstrating with hard data that readers understand the difference between real content and — let’s call it what it is — advertising and they quickly abandon it.

The bottom line of Tony’s data is bad news for cynical publishers who have tried to manipulate readers with link-bait headlines and lists, and who are trying to pull the wool over advertisers’ eyes by selling them link-bait listicles and so-called native advertising. Certain emperors have no clothes. The readers know it. The advertisers will wake up and realize it.

But that’s the bad news.

Where we should turn the discussion next is to what the right metrics for media should be. As they say, you get what you measure. So what should we measure? How do we create positive feedback loops that improve the news, not degrade it as unique users, pageviews, and other relics of mass media have done?

I’ll start with the most important and most difficult thing to measure: outcomes. Were people more informed because of what we gave them? Did they accomplish what they wanted as individuals (Sally got new health insurance and saved money) or as communities (Riverdale cleaned up that messy park)? I just had breakfast with Robert Rosenthal of the Center for Investigative Reporting and he told me they start the process of reporting by considering impact and they end by trying to measure it. Why deal in bad proxies for good journalism, based on popularity, when we could get to the reason journalism should exist: to improve the world?

In his book News: A User’s Manual, Alain de Botton says that news has “the power to assemble the picture that citizens end up having of one another; the power to dictate what our idea of ‘other people’ will be like; the power to invent a nation in our imaginations.” And it has the power to help us get there. (Many more quotes in my post about the book, here.) Mark Zuckerberg says that platforms, including news, should offer communities “elegant organization.” These are higher aspirations than mere exposure.

On a tour of technology companies in Silicon Valley a few weeks ago with my dean, we talked about metrics and found different measurements being used for different platforms with different goals. Ev Williams’ Medium values total time spent reading. That is appropriate for a platform that wants to get people to explore ideas in depth — and I find I’m spending more time there reading more posts; it’s working.

Attention, in the form of time spent, is used by many in media as a measure of engagement. But that’s not always the case. Attention can also be another egocentric media metric: how many people come to look at my stuff; how many pages of my stuff do they look at; how much time do they spend with my stuff? No, sometimes, the less time spent the better. What if news were more efficient? Sometimes, spending less time to get what I want is the right metric. That metric doesn’t serve the old media business model of delivering as many eyeballs to as many ads as possible. That is why Yahoo shifted from — in the words of cofounder Jerry Yang — getting you in and out with the answer you needed as quickly as possible to instead trying to bombard you with content and keep you around as long as possible to show you as many ads as possible. Attention, in the wrong hands, can also be a corrupting metric.

Cir.ca has a fascinating metric: follows. When a reader follows a story, she is telling Cir.ca, “Please bother me and let me know when something new happens here.” That is a measure of true interest.

Similarly, Flipboard keeps track of how many people subscribe to a publication — and even to an advertiser’s publication. It also watches what people “flip” or save to read later, which strikes me as a much better indication of interest than sharing.

Google has long valued links as a digital version of citation. That has served search well. Google News also uses citations to try to infer which news organization created or is staying on top of a story — if everyone writing about Walter Reed Army Medical Center quotes the Washington Post then there’s a good chance it’s the Post’s story.

Repost.US and YouTube and now Getty Images track embeds — how many people truly want to share a video or an article because they repost it in their own space on the web. The problem with just “liking” or “sharing” on Twitter and Facebook is that there turns out to be no cost for those transactions; it’s too easy to just keep passing things on. Embedding uses my space and affected my reputation with you. I would like to see more such higher friction means of sharing that really do impute engagement.

What is engagement? It’s likely not one measure of one method of interacting with content. It could be that I spend time with something, that I interact with it or the people gathered around it (though don’t we know that comments are no indication of quality), that I save it, that I take action based on it.

We want to find good proxies for engagement in the hopes that they will lead us to indications of quality, which in turn should tell us something about the authority of the creator and the trust the public has in her. None of these is easy to measure, like “likes.”

Another word for engagement is relationship. I have been arguing that we in news should stop seeing ourselves as content factories and start seeing ourselves as members of our communities who are in the relationship business, who use what we know about people to better serve them. Thus, I ask media companies how many relationships they have with the people they serve and what they know about them — what signals they have, enabling them to improve relevance and thus value and often impact. Those are metrics that start with the public rather than with media. Those are metrics that matter.

Selling ads by time, not space

I just saw some mind-bending work Chartbeat is about to release about measuring the time users spend exposed to an ad online.

As background, to quote Chartbeat CEO Tony Haile: “Chartbeat monitors activity by checking in with users every second and looking for signals (mouse movement, key strokes, etc) that show they are actively consuming the content in front of them. This means they can measure how long readers spend actively engaged on a page and what parts they’re reading. Because of this Chartbeat knows how long are actively reading while an ad is in view — both for an average user and the cumulative time of all users.” Chartbeat then did some internal research that found high correlation between engaged time exposed and a user’s ability to recall the advertiser’s brand and message. This has many implications:

* Measured this way, ads that appear down alongside the middle of a story turn out to be more valuable than the supposedly premium banners at the top of the page. That’s because people quickly scroll past those banners and all the big hair on the top of the page — logos, promos, and all that — to get to the substance of an article, where they spend time. So inventory that was undervalued becomes more valuable.

* Chartbeat suggests this means that quality content that engages people longer yields better ad performance. That, they say, would be a good thing for better content makers everywhere.

* Now web publishers can sell time like broadcasters — only this is assured exposure time. Advertisers like buying time. Will this make them more comfortable with buying on the web?

* I think this enables publishers to take on some risk for advertisers — guaranteeing them assured exposure time — thus increasing the value of what they sell.

* I wonder whether this spells trouble for the big-ass ads and takeovers we users try to escape as quickly as possible.

* I also wonder whether this spells trouble for the slideshows and other gimmicks that pump page views without increasing time spent exposed to an ad.

* I’d like to think this opens opportunities to find new value in ads next to videos and games and also — this could be important — mobile pages (though don’t think that mobile’s value will come from exposure to messaging; it will still come from knowing people and serving them relevance and value). The longer we spend on a page, the longer we see the ad, the more valuable the ad should be, right?

* I can only hope that this is another nail in the coffin of the dangerous, old-media-like metrics of unique users and pageviews. Engagement will matter more.

A sample report on an ad location:

Screenshot 2013-05-15 at 8.20.04 PM

Those who declare advertising dead are Mark-Twaining-it, I think. There are still many things to learn to find more effectiveness and value in advertising online. This is just one lesson. I say the real value of the net and mobile is in relationships: in learning more about people by delivering them more value so we can be trusted to deliver them greater relevance and value and, in turn, extract greater value from the interaction. More on that later….