Measure this

The means of measurement used by advertisers for every other medium — newspaper, magazine, radio, TV, and no online — will not — will never — work in the world of citizens and distributed media. That is why we must create our own measurement standards.

To get apples-to-apples numbers for those other, older, major media, advertisers rely on allegedly representative samples.

But you can never get a sample big enough to deal with the mass of niches.

Hell, the samples aren’t big enough to deal with local online newspaper sites. The Online Publishers Association just released a study that found various means of measuring those sites disagree drastically:

Differences arose between the two primary methodologies, surveys and panels….

The paper analyzes data from five services. Firms conducting panel research include comScore Media Metrix and Nielsen//NetRatings-MegaView Local. Firms measuring local audience through a combination of online, phone and postal mail surveys include Nielsen//NetRatings @Plan (online); Scarborough Research (phone and mail); and The Media Audit (phone).

When data from the two forms of collection were analyzed, survey-based methodologies, on average, reported 70 percent higher visitor numbers than panel-based research.

An example cited in the study looked at the number of visitors to Visitor data differed by one million between two services.

When I was at Advance, we found that these sampling methodologies would find no audience in some markets that we knew from our server stats were actually much bigger than other markets they did measure. They simply didn’t have enough people in Alabama.

Well, they’ll never get enough knitters to measure the knitting bloggers. They can measure a few of the biggest bloggers. But that’s not what this medium is all about. It is about, as someone said at my Web 2.0 ad panel, the “big butt” attached to the fabled long tail of passionate niches that add up to a mass far bigger than the biggest bloggers. So we need to be able to add them up.

This is why it is doubly important for us in this world to create and use our own means of measurement. I’m talking with some folks who are better at getting things done than I am — and working with Burst Media‘s coincidentally named Jarvis Coffin to set up a trade group — to work on open-source collection and reporting.

This isn’t just about collecting and verifying audience and pageview numbers — and demographics and behavior — though all that is important.

This is also about collecting data that can be collected only in this medium of the people and gives us unique value: authority, influence, conversation-starting, relationships, loyalty, engagement.

And this is about additional data that cuts across sites — from the likes of Technorati, Icerocket, Blogpulse — and how all this data will be munged together by various parties doing their own analytics.

So, in the end, when an advertiser wants to reach top food influencers they’ll be able to do so through influential food bloggers … and those bloggers will be able to recognize their value as well.

But it won’t happen through the survey or panel research that have become advertisers’ crutch.

  • If you are looking for a new measurement system you might want to look at the one that caused advertisers to invest $52 billion of their ad budgets in 2004. That’s more money than broadcast TV got, more than newspapers, 4 times more than glossy magazines.

    The medium is direct mail (a subset of direct marketing) and the measurement system is “accountable results.” It’s a tough yardstick… when you don’t measure up you have to change and quickly improve your product and your methods. And the only way it works is if you stay very, very close to the customer and the front lines in business.

    But it’s the measurement system that advertisers (the folks who sign the checks) want. Ask them… they’ll tell you.

    I’m not in the direct marketing industry but I’ve been amazed at how they have been able to go from virtually nothing in the 70s to such a huge share of the advertising spend today. I think part of their secret is in their measurement system.

  • As this medium gets measurement better defined (and it is certainly deeply in the works), there is a fantastic opportunity for it to move upstream and help provide some grounding for other numbers. That is part of the real promise of blog/consumer-voice measurement, and why we see a lot of development and investment in this space.

    In particular, the notion of engagement (as you mentioned) could prove the single most interesting measurement point.

    Good post. Thanks.

  • Jarvis & Jarvis. Great combination!

    Can I help?


  • Tim Hanlon

    Amen brother!

    Nielsen can’t keep up with the today of television (hundreds of digital channels – including every single HD channel out there, on-demand, time-shifted DVR, interactive screens), let alone the tomorrow (broadband-enabled video that “crosses the transom” into TV environments, etc.).

    Arbitron still uses *paper diaries* to measure listening *recall* for radio. And that’s supposed to also keep up with HD multicasts, satellite radio, mobile/wireless audio, etc.??

    We’re already way behind on how people consume traditional media; the shift to more census-oriented measurement for those and future vehicles of communication can’t happen soon enough.

    Media companies and ad agency professionals who continue to hide behind the blunt/dull media measurement instruments of yesteryear do so at their own peril in a world that will fast become far more transparent and data-centric.

  • Matt Galloway at the basement blog ( has been writing amazing stuff on how to measure “influence” and its economic values among blogs. The methodology is mostly based on the word-of-mouth approach and is definitely worth taking a look at.

  • WOMMA is a non-profit that has a large committee of major researchers working on standardardizing measurement terminology, with a strong focus on blogs. We’ve come a long way to building a framework around measuring concepts like influence, reputation, etc. Lots of background and free presentations at and

  • Very interesting Jeff, I am interested to hear more about the trade association you are thinking about for open-source measurement. Keep me posted.

  • You may want to take a look at a White Paper I just released.

    It deals with objective measurement of influence on public debate.

    In the academic world citation indexes have long been the established way of objectively measuring the influence of journals, universities and researchers.

    The same methodology can be applied to information in the public arena (e.g. the Internet)

    A short summary of the paper can be found at the link below where the whole paper can also be downloaded in .pdf format.

  • Pingback: Metrics, successes, & flaming disasters in online marketing()

  • Pingback: theory.isthereason » Today’s Links: Losing our ownership of ideas()

  • I would like to send you some information in respect to what ARF is doing, and what you are writing about. If you notice this, please email me at [email protected]