The problem with measurement

Measurement is holding back so much progress online — both because there’s not enough of it and because what there is of it is too often tracking the wrong things. Without good measurement, we won’t get money and without money we won’t be able to sustain so much of what we want to do online.

First, about measuring the wrong things. Nielsen just switched from measuring page views to time and that’s a step in the right direction — about a year and a half too late. It’s finally acknowledged that the pageview is dead thanks to dynamic content that makes act like shows. Heather Green talks about how time is also a meaningless measure, though, because we simply spend more time with some applications (e.g., IM) than others (e.g., content). Time is not a good proxy for engagement. This gets much more complicated when content and functionality becomes widgetized and distributed. There is no means of measuring traffic/attention/engagement for widgets. And this also means that the site itself becomes obsolete as content and functionality are distributed anywhere and everywhere; I predict that we’ll soon see brands that essentially don’t have sites but become huge because they are distributed (think apps on Facebook that have no home page). I’ve often said that Google’s audience is many times what is reported because Google distributes itself as widgets — ads, maps, feeds. Google doesn’t care because it’s measuring only one thing: revenue. And Google knows everything it needs to know. We don’t.

Now about what’s not being measured. There’s a very long list of applications — RSS, widgets, mobile, apps — and kinds of content — video, podcasts — but also of new sorts of measurements — such as influence, meme-starting, involvement, creation, engagement, popularity — that aren’t even being tackled. And there are new dimensions that need to be explored, such as measuring a person’s trust, influence, or even fame across many platforms, sites, applications, and so on.

If it’s not measured, advertisers won’t buy it. And if advertisers won’t buy it, companies won’t build it. And that’s the problem.

Well, but that’s only somewhat true. Big, old companies won’t build it. New inventors will and that gives them the headstart they need. YouTube, Facebook, Twitter, Pownce, and so many innovations started without measurement — or evenue. But they had to have the courage of their vision and enough money to make it big first. What else has died just because conservative advertisers and agencies and their outmoded measurement companies — who are still using sample panels that simply will not work in a nichefied world — hadn’t caught up with them?

I’ve suggested before the need for a measurement summit. Maybe that’s one way to tackle this. Or perhaps we need to find ways to better educate advertisers (but we’ll never educate them out of wanting metrics). Or perhaps this is something academics should take on with foundation help because it will support invention: measure it and they will come.

: LATER: More from Heather.

  • http://deleted Tansley – addendum

    Truly interesting topic, Jeff. The problem isn’t new: television has had the same problem for decades, particularly since the advent of the remote – Neilsen reflects viewing numbers, but not who is clicking the mute button for a kitchen trip during commercial breaks, and who isn’t.

    Click-thoughs only measure directed interest, not exposure to memes or names. Page views don’t take into account pop-up blockers. I once used a firewall (Agnitum) that literally blocked ALL flash from my surfing. (No, I don’t own stock in them, btw.) For a few weeks, my surfing was pristine, quiet, quick-loading and virtually ad-free….BUT (and this was a big one) it also blocked a lot of java scripting…and I literally found myself unable to get into some sites I WANTED to visit, since they relied on java triggers. So…I uninstalled Agnitum and went back to McAfee. Live and learn.

    And no, TIME is really no adequate measure, either. I can spend hours at a blog and never notice the marginals.

    It will be interesting to see WHAT will prove an adequate unit of measurement to entice advertisers. Of course, there’s always response-polling (how did you learn about our site/product/service? – choose one of the following…) but a lot of people never bother with that.

    Keep us posted on this one. My paper is just getting into the web in a big way as we speak…

  • Greg0658

    Easy. Labor Based Society. Get off the monetary base.

    Get up, go to work, get off work, then eat drink and be merry.

  • http://www.linkedin.com/in/jurup Jacob

    Jon Lund, director of the Danish Media Online Industry, had an interesting blogpost on this issue yesterday (http://newmediatrends.fdim.dk/2007/07/audience-measurement-giant-leap-forward-now-were-in-the-race.html). Apparently, they combine the raw statistics with monthly surveys by a polling institute to get qualitative data too.

  • Pingback: ethority blog » Messbarkeit im Netz

  • Cooler Heads

    I specialize in measurement of things like behavior (difficult to be sure). Yes, you should have a measurement. And you should respect the power of measurement rather than seeing it as some nasty stepchild that you can’t quite shake out of your house.

    The webworld is full of smart people, and someone needs some kind of algorithm to capture–as well as possible–the real behavior of people reading blogs, etc. My suggestion, have the summit. But first ask a couple of really smart measurement types to do some qualitative research on web behavior so that the summiteers can begin to form items for large-scale measurement. That’s what advertisers want, large-scale measurement.

    I think this is fascinating, and whoever comes up with a good method could make a lot of money.

  • David

    Jeff,

    There have been web measurement conferences for years, most notably the Emetrics Summit. There are two bodies responsible for developing web metrics standards: the Interactive Advertising Bureau (IAB) and the Web Analytics Association (WAA). Both have been around for many years.

    There are plenty of ways to measure RSS, widgets, P2P downloads, and every other means of content distribution – any server call can theoretically be tracked. The tricky part is integrating the data collected for each of these distribution models and determining their relative value for both content producers and advertisers.

    Additionally I would say that there are many others who are far more qualified than you to speak with authority on this subject. I would recommend doing some research before you post anything further on this subject.

  • http://www.buzzmachine.com Jeff Jarvis

    Well, David,
    The IAB has been behind, too.
    RSS doesn’t enable cookies and clickthrough measurement without redirects.
    Downloads don’t ping back to verify viewership.
    P2P downloads likewise.
    Until recently, with podcasts, neither allowed dynamic and targeted serving of ads.
    And there is so much more to measure, like who starts memes and their biorhythms.
    I’ve researched plenty, every time an advertiser has whined that what he wants measured isn’t measured and so he won’t spend him money; he likes going to upfront parties instead.
    We can measure much better. Wouldn’t you agree to that?
    And why don’t we have a civilized discussion about this. Unless you prefer the nya-nya-nya tone. Up to you.

  • David

    Jeff,

    Sorry for the tone in my previous reply – I was disappointed that you didn’t mention some of the people who have been trying to bridge the (admittedly) vast gaps that exist in the measurement field and by your pessimism, but perhaps this is just another testament to a lack of success to date. FWIW you’ll probably find that we don’t disagree much if at all.

    Anyway, there’s obviously a tremendous disconnect between the brand advertisers/big agencies and the internet ad community. Part of the problem IMO is that agency media buyers simply don’t understand the medium – they are fixated on the equivalent of TV reach and frequency metrics that Nielsen NetRatings and ComScore provide (the page view metric is/was really more of a–flawed–proxy for available ad inventory). At least as far as I can tell, they don’t seem to realize that these numbers may not have much of anything to do with how many impressions their banner ads receive. Maybe it’s an inability or unwillingness to recognize that the web is not–and will never be–a mass medium? I don’t really know, but if you’re a big agency with massive infrastructure and overhead–much of which can eventually be replaced by computers–you probably don’t want to consider the possibility that, with the possible exception of outdoor, etc., the future might not look very bright for effectively reaching mass audiences.

    Nice (if long) summary here –
    http://blog.jimnovo.com/2007/06/12/media-measurement/#behave

    You’re right that the IAB has been (very) behind, particularly given how long it took them to address the disparities between the panel-based measurement services–specifically Nielsen NetRatings and ComScore–and internal server logs. Inevitably one or the other will mention the effects of cookie deletions (which can vary widely from site to site) and non-US visitors, but I also think as you do that there are methodological problems, which are probably too numerous to go into here.

    As you wrote in your original post, it’s really not clear whether a sample panel, regardless of size, can produce accurate projections when the audiences are so fragmented. It’s difficult enough for small websites, let alone RSS feeds, widgets, and facebook applications. None of these major measurement firms IMO have shown that they are capable of moving proactively to address the emergence of these technologies.

    Incidentally I think Facebook has opened up a new world in terms of identifying influencers because application developers could establish betweenness and other centrality measures based on how a given application (or some social function present within it) is adopted by other nodes across multiple networks.

  • Pingback: The Interactive » New media needs new measurement

  • Pingback: Social media index « Technobabble 2.0

  • Pingback: terrainnova.org

  • Pingback: Is Link Authority Dead (Dying)? - Internet Insider Report

  • Pingback: carlo beccaria - blog / links for 2007-07-20

  • Pingback: The social media revolution (in 15 minutes)

  • http://kdpaine.blogs.com Katie Paine

    You were invited to a measurement summit and haven’t yet responded, so it’s happening anyway. I think your readers should know that there IS a measurement summit happening in Portsmouth, NH October 4th, 2007. http://www.instituteforpr.org/education/som_program_content

  • Pingback: Metrics in Conversational and Community Marketing

  • Pingback: Is Link Authority Dead (Dying)? | SEO Ranking, Marketing and Tactic

  • Pingback: Is Link Authority Dead (Dying)? | Corporate Internet Reputation Management

  • Pingback: Is Link Authority Dead (Dying)? | Press Release -Reputation Management

  • Pingback: SEOForumz Blog Site -Learn SEO and more.. » Blog Archive » Is Link Authority Dead (Dying)?

  • Pingback: Is Link Authority Dead (Dying)? | We Do Web: News, Auction, Seo, Security

  • Pingback: CustomScoop | Measuring the many facets of social media