The need for a measurement summit

Comscore and Federated Media (which sells some ads on this blog) have teamed up to try to improve measurement in the long tail of social and niche media online. And that’s good.

Except I argue that the panel means of measurement is doomed to miserable failure in the mass of niches. You cannot possible build a panel large and varied enough to get reliable measurement of the audience and traffic of millions — even thousands — of sites, especially when we get the means to tie together lots of those small sites into networks.

What I hope they do is honestly and harshly look at their stats from their panel versus the server stats of the sites — especially the smaller sites, not the much-easier-to-measure big boys like Digg and BoingBoing — and realize that the panel just doesn’t work.

What we need, I’ve long argued, is standard metrics reported from the sites’ servers or from snoopers on pages and verified by a service such as Comscore or Nielsen. Old methods will not work in this new world. The same goes for Nielsen, which is buying the rest of Netratings.

And whilel we’re at it, let’s figure out the new measurements that capture the unique value of this new medium: authority, speed, connectedness… The page view is dead.

I think it’s time for a measurement summit: Bring together the measurement companies, the advertisers and their agencies (buyers), the sites’ reps (sellers), the media sites, and technology companies and let’s hammer out some standards and methods for measurement. This will only work if we have open standards with analytics (like Comscore and Nielsen) building value atop that common data. Otherwise, we end up in a world that will continue to confuse and scare advertisers — and their money — away.

  • http://www.jaybryant.com Jay Bryant

    Thanks for the post, I would agree that we need to bring the measurement business out into the open with standards that are industry supported. Let me know if you need space for the summit.

    Jay Bryant

  • James Lewin

    Jeff

    Worrying too much about page views, authority, speed, etc, statistics seems to be horseless-carriage thinking.

    If a company is advertising on the web, they need to be measuring the results that campaigns deliver. This is something companies can and should be doing now, but few are.

    On the other hand, it will be a long time before there’s a consensus on measurement of penetration, etc.

  • Pingback: Andy Dickinson.net » Blog Archive » Quickies: Measurement and web video

  • http://www.madmedia.ca heri

    i think what we need actually is a buzz tracker, or a tool which can track audience for a niche site, where they are coming from, what users are asking for, measuring their potential interest etc. I agree that panels and old metrics don’t work any more.

  • http://fannick.blogspot.com Glenn Fannick

    Dow Jones’s Factiva group is one of those companies actively investigating better measurement approaches. We held a roundtable discussion in Palo Alto last year to start some ideas flowing. (A white paper is forthcoming) We want to do more in this area in 07. It’s badly needed.

  • Pingback: Chip Griffin: Pardon the Disruption

  • http://pegasusnews.com Mike Orren

    While I agree with James Lewin’s sentiment above, the advertisers want to see traffic numbers — and breaking them of that habit will take some time.

    I’d absolutely volunteer to participate from the niche/local side. This undermeasurement problem is particularly acute on local sites.

  • Matt Sollars

    Such a discussion is even more important with politicians moving their campaigns online. Hopefully, we can develop a way to judge a candidate’s online popularity a bit more accurately, and without the apparent spin, than stories like this one at politico.com yesterday that declared Hillary the most popular online Dem. She may well be, but the numbers used in the story aren’t all that convincing.

    If we are to make judgments on candidates based on online metrics, then we really need to understand what those metrics are and what they mean.

  • http://dl.tv Jim Louderback

    Absolutely. This is one of the biggest problems we have today for video on the web, along with podcasts, as much of it is downloaded, vs. streamed, and impossible to track.

    I’m there!

  • http://insights20.com Olivier

    Hi Jeff,

    I think that the Summit you are looking for already exists. You might want to check the ESOMAR Worldwide Multi Media Measurement Summit (WM3) which will take place in Dublin in June and gather researchers and all media to discuss about measurement challenges (it was held last year in Shanghai).

    I believe that the Program Committee is still receiving submissions for papers…

    Olivier

  • http://www.sergetheconcierge.com Serge Lescouarnec

    Jeff

    It seems it is hard to get a clear picture.

    For my blog ‘Serge the Concierge’, I now use the Stats provided by Feedburner which give a good picture but not a perfect one.

    Maybe this is due to different companies and services using different methods.

    As for advertising, I think it works better if the ads are closely related to the content and audience of the site regardless of the audience size and also of a certain quality.

    Some sites run the risk of looking like billboards with a few stories.

    My few cents.

    Serge
    Blog:
    http://www.sergetheconcierge.com
    Biz:
    http://www.njconcierges.com

  • http://kdpaine.blogs.com Katie Delahaye Paine

    As someone who’s been measuring CGM for a decade, I’d love to be at a Blog Measurement Summit. A decade ago we were having this same conversation at a summit about how to measure web pages — and unique visitors was the best we could come up with at the time. Now at least, there are real business outcome metrics available. It would be great if we didn’t have to wait ten years for good CGM metrics.
    And I totally agree with the importance of connectedness, and level of conversation, but too many in the agency world are still obsessed with quantity of eyeballs, and the loudness of their screams, rather than on the actual business outcomes.

  • http://www.telegraph.co.uk edward roussel

    Greetings,

    The page impression is far from dead – just speak to the media buying agencies who still consider it to be the best available measure of volume of traffic (as oppose to number of readers or frequency of visits).

    There will never be a consensus about a single metric. Yes, ABC has now made uniques their mandatory measure; but that doesn’t mean that other measures matter less. The significance of a particular metric depends on what it is that an advertiser (or layman) is trying to measure. Indeed, ABC offers this useful guide: http://www.abce.org.uk/

    Boils down to this:

    Use uniques if you want to measure “audience reach”;
    Use page impressions to check out “volume” of traffic;
    Use visits to gauge “frequency”.

    For most people, the above three measures provide more than enough information about the performance of a website.

    Edward

  • http://toddwarfel.com Todd Zaki Warfel

    This is definitely needed. With the influx of RIAs and the slow death of the page, metrics is changing. Most of our clients are trying to use things like Google Analytics, WebTrends, FireClick, and Omniture. None of these work great for RIA situations. We have done some customer tracking with Flash-based sites that track individual MCs. So, we can track states, which is more granular, but this is a different model than what the industry has been used to in the past.

  • http://perfcap.blogspot.com Adrian Cockcroft

    Edward says:
    >Use uniques if you want to measure “audience reach’’;
    >Use page impressions to check out “volume’’ of traffic;
    >Use visits to gauge “frequency’’.

    I’d add one more metric – stickiness or time spent on a site, since a fleeting ad impression is worth less than one that gets a lot of exposure. This could help capture some of the AJAX based sites, where the ad in a surrounding frame gets a lot of viewing while the rest of the page is being modified.

    If you do get a summit together, let me know and I’ll do what I can to get eBay involved (I’m in the research labs at eBay).

  • http://www.blogbusinesssummit.com Steve Broback

    The blog business summit team would be interested in the organizing/hosting the “blog measurement summit.” I know several good potential venues. Jeff, ping me if you want to discuss. My email address is attached to this post.

  • Pingback: hey jeff jarvis » Instant Cognition » web analytics

  • http://www.visioactive.com Ian Houston

    Hi Jeff,

    Interesting post. A subject near and dear to my heart and one I’ve spent a fair amount of personal research on the past couple years.

    The organization you should check out is the Web Analytics Association which enjoys participation from a major portion of the major analytics vendors. There is also an associated conference called Emetrics Summit where myself and others have presented directly about this topic as well as Web 2.0 measurement issues a whole.

    If you are interested in looking at a somewhat simple example of RSS event based tracking I recommend the book Web Site Measurement Hacks by Eric Peterson where I contributed a hack with a basic perl example using the “snoopers” you mention above.

    My point in all this is there is a fairly substantial community out there already talking, meeting and working on these issues that includes a major percentage of companies that build the tools you would use to do this tracking, the people who use them and a “niche” of bloggers. Perhaps we should all get together for coffee sometime. Maybe in May, in San Fransisco during the Emetrics Summit?

    Best regards,

    -Ian

  • http://web-strategist.com Jeremiah Owyang

    Jeff

    Many of our clients at PodTech have a need to measure success (esp since corporations that are hesitant to adopt).

    I’ve spent quite a bit of time researching this information (and Clint Ivy points to me)

    Please check out some of my thoughts

    Defining Engagement
    http://www.web-strategist.com/blog/2007/02/01/defining-engagement/

    Discussions with Nielsen Buzz Metrics CEO
    http://www.web-strategist.com/blog/2007/01/26/social-media-measurement-industry/

    I worked with Factiva/Dow Jones to run a workshop with some of the top minds in social media and the topic, an invite should have been extended to you:
    http://www.web-strategist.com/blog/2006/12/06/factiva-social-media-roundtable-helps-to-answer-what-should-we-measure/

    You can see all my thoughts that are tagged “Social Media Measurement”
    http://www.web-strategist.com/blog/category/social-media-measurement/

    I was at Hitachi, and running the Social Media program, it was my job to measure success and impact. I did it using a variety of tools as well as a vendor, so I come from experience in the ‘trenches’.

    I’m interested in learning how to participate at your event.

    In the end, I hope to be an influencer, as I’ll be recommendations to our clients, both in person as well as from my blog.

  • Pingback: Time for a new PageRank? | terrainnova.org

  • Pingback: "Community Project: Roundtable on Social Media Measurement" from Pro PR

  • Pingback: Measurement Summit? Count me in! - One By One Media