Two people I respect tweaked me — one in a post, one in email (both with a business interest in the matter) — for what I wrote about the comScore kerfluffle (or efforts to start one). I think they misread me but that probably means I miswrote it, so let me restate to be clear:
First, I am delighted that we have the comScore research. I think this is incredibly important in the growth of blogs as a business — for those who want it to be a business — for this states their metrix in terms that advertisers will respect and understand… and buy. Remember that I pushed very hard for such research to be undertaken at Bloggercon II.
Second, the things in the results that don’t sound sensible can probably be corrected with improved methodology and I hope that comScore is smart enough to call upon the expertise of this open world to help them improve that methodology. Fred asks for Jason to hand over his server logs to help them and he asks for time and he’s right:
The issue with panel research is you need time to develop the statistical algorithms that weight the panel data correctly before you scale the numbers. And you need a very good dictionary of the domains you track. These take time to nail down. Clearly Comscore hasn’t nailed it yet, but they never said they had.
But third, even with the best study, I stand by my point that panel research — as valuable and necessary as it is for advertising to be able to judge an even playing field of properties — is at the end of the day still bullshit. Fred argues:
Jeff is so wrong about this that I find myself shaking my head on this one.
First, the problem with panel research of old is small sample sizes. The panels that have been used for decades in old media are almost always less than 100,000 people primarily because it is cost prohibitive to collect data from a larger panel. Clearly that is way too small for accurate measurement.
But Comscore invented the concept of a “megapanel” in 1999 and is currently measuring over 2 million unique Internet users. That’s the beauty of the Internet, you can measure online at a scale unimaginable offline. At that kind of scale, panel research is not only accurate, its amazingly accurate. As far back as 2001, Comscore was able to predict a missed quarter for Amazon.com. And the panel size and the technology have improved significantly since then.
Well, I spent years in publishing and then the internet seeing how wrong panel research could be. People magazine allegedly had
eight readers per copy according to panel research. On the face of it, that’s absurd. But it determined the readership and thus cost per thousand and thus ad rates for People. And that paid my salary. But it was bullshit. And everybody knew it.
Online, I saw these big panel studies rate some of my smaller sites as huge and some of my biggest sites and nonexistent. And the reason for that, clearly, was that the sample didn’t have people clicking on mice in Ratbutt, Alabama. The problem is that when you measure small things — and blogs, individual blogs, are still very small — the effectiveness of a sample panel will decrease exponentially and the impact of one reader or one missing reader can be amplified to ludicrous extremes.
My fourth point is that, yes, I agree with Fred that Jason should take a deep breath and get over it and others who nitpicked the study should take it in the context of all such studies: They are all bullshit, so untie those knotted knickers and move on.
For, yes, this is an important study. It shows that blogs are big and growing. It shows the relative size of their audiences against other media properties and makes them real. It starts to give advertisers (and us) the flavor for the unmedium — our interests, our habits (e.g., political blogs get more traffic from smaller audiences), our demographics, and all that.
So what I’m really saying — though, clearly, I said it badly — is that the details and the nitpicking and the fact that it’s just bullshit doesn’t matter and so it’s not worth fighting over, especially if it can be improved. This is what advertisers need. This is what will make them buy.
My fifth and final point is that we would be foolish to stop at this kind of syndicated research as the basis of building blogs as a business, for — just like Google AdSense — it devalues and misjudges us. We have greater value in our relationships and influence and we need to find ways to measure that. No, that is not a substitute for the basic audience metrics advertisers demand. But it will prove our higher value and we can then get paid for that higher value and that is a good thing.
