Numbers don’t joke

Jason Calacanis is having proper conniptions over the comScore marketing study on blogs released this week. Fred Wilson and Heather Green have moments of dubious doubt themselves. Krucoff, says Jason, nails it and quizzes Rick Bruner, who helped on the study itself, as Rick tries to answer questions on the methodology on his blog. I leave it to you to follow the links above to the specifics.

My first reaction is that all this shows how messed up panel research is. This is the method used by Nielsen et al to measure TV and radio and print readership — affecting billions of ad dollars — and it is and always has been relative bullshit. That’s why advertisers buy it, though — because it is relative, because they can compare this magazine to that magazine on the same sheet. But it’s all based on a small and only allegedly representative sample of people. It’s meaningless. When I worked on magazines that allegedly had eight readers per copy — damned dogeared, they were — we benefited from this relative bullshit. But when I came online, we could measure the bull ourselves: We could compare our server and cookie stats with what the panel research told us and we could tell when they didn’t have any panel members in entire states. Panel research is a novel in numbers.

My second reaction is, however, that blogs need some sort of numbers advertisers will buy. I thought it was a good idea to try to get that research and, as I read that link, I see that this is partly my fault: At the blogging business session I emceed at Bloggercon II, I emphasized the need to feed advertisers their metrics. So it’s a damned shame that this research is raising such eyebrows.

My third reaction is that we should be creating our own meaningful metrics. Bloggers who care about making business at blogging (and let’s remember: that’s only some of us) should be agreeing on cookies and also on new means of measurement and new things to measure.

This isn’t as simple and stupid as an a-list or a panel or page-views or eyeballs. This is a much richer thing, this unmedium of ours, and it needs much smarter measurement. See Mary Hodder’s napkin notes for just some of the means of measuring blogs’ popularity and appeal; I can think of many more.

Advertisers are screaming for proof of “engagement” these days and while silly, inky magazines are trying to “engage” with flashing ads on paper, we flash without trying. We engage or die. We live by relationships and trust — more fave ad words. We have influence — yet another fave word. We need to measure and report all that.

Instead, we’re futzing and fussing and fuming over the few numbers we have and giving advertisers another excuse to ignore us. Arrgh.

comScore should reveal much more about its study so that bloggers can poke at it and so the crowd — the wise crowd — can help improve the methodology and ferret out what makes sense and what doesn’t — and let’s remember that these numbers do show that blogs are a thing that should not be ignored. If something looks odd, explain it or explain why you can’t.

And we should start finding new ways to measure our real value — and that’s not about continuing to chase the big numbers that are so old-media and it’s not about continuing to value relative bullshit. We need to find the numbers that count.

: Oh, and while we’re at it, can somebody point me to exactly what the oft-quoted and bragged-about Alexa numbers are really based on? Does there data come from the toolbar still? Do you know a single soul who actually uses that toolbar? How big is their sample? How representative?

Every one of these services that now tout numbers should be transparent about methodology and sources and scale. It’s late, so I’m not going to go looking now. But if you can point me to such disclosure at Blogpulse, Technorati, Bloglines, et al, please leave a link in the comments and let’s start by finding the best of breed in transparency.

  • http://epeus.blogspot.com Kevin Marks

    Hi Jeff,
    at Technorati we’ve always disclosed our methodology.
    We count distinct links currently on the front page of a blog or feed for our rankings.
    This means linking to the same blog multiple times counts once – one blog, one vote. As they move off the front page, they stop being counted.

    By clicking on the little bubble icons, you can see the links we are counting.

    I’m not claiming this method is perfect; there is certainly room to experiment with other techniques, but I do think it is transparent.

  • http://www.bloodandtreasure.com Noel Guinane

    Nice try, Rick. Shame it looks like a beautiful bronze sculpture of Venus de Milo but smells as though it were cast in cow manure with a lick of gold paint quickly applied for surface sheen!

  • http://sethf.com/ Seth Finkelstein

    Cynical question: Has any business survey paid for by clients, ever shown those clients in less than the most glowing light? (I feel like I’m asking if a man has ever bitten a dog – maybe it’s happening somewhere, sometime, making news).

  • Jim Dermitt

    Numbers are serious. Just look at Mickey D’s. The sign says billions and billions served. So people see that and think, hmmm, must be a heck of a good burger we better stop and try one. The local guy has only seved thousands and thousands. You can’t get very far in the physical world or discover much worth discovery by limiting your reasoning to only the logical processes of arithmetic. Big numbers are impressive and everything counts in small amounts. A spider doesn’t do any calculations, it just builds webs based on random data and environmental conditions. The process is complex but it is not ruled by numbers. A spider ain’t digital folks.

  • Jim Dermitt

    Numb-ers and news. HEADLINE: A $50 Million Price Tag, and Still a Fixer-Upper? Welcome to the numberhood.

  • Jim Dermitt

    Jeff, Maybe you can just use numbers to ID all of your posts.

  • Jim Dermitt

    One more time to kill, one more time to feel. Whatever I do, it’s never enough.

  • http://spap-oop.blogspot.com Tish G

    Well, Jeff…just goes to show that as much as anybody wants to claim this is “new media” we are still stuck in the mindset of old media. Even the idea of Denton wanting to build an on line “media empire” is old media think. And that he’s doing it in the manner that Rupert Murdock did–by upping the hottie factor–says that using the old tools to build a media empire work just as well on line as they do in print. And that the even older way of skewing the stats can help too. I just love Calacanis’ chutzpah.

  • http://www.randomculture.com John

    Alexa data is still determined by toolbar data… What’s more, according to the “About the Traffic Rankings” page on Alexa, you’re only seeing data for IE browser traffic, and Windows operating systems. Firefox could easily account for 10, 20, 30 or more percent of some sites’ traffic… and wouldn’t show up in Alexa.

  • http://cellar.org/iotd.php Undertoad

    I knew it. I toldjaso! sportsbybrooks.com? It’s not even a blog!

  • Jim Dermitt

    Do the Star Trek logs count?

  • http://www.phillyfuture.org Karl

    Jeff, I think you’re on the right track here.

    So why the change in heart?

  • http://ambientirony.mu.nu Pixy Misa

    I installed the Alexa Toolbar – primarily so I could see my Alexa stats (currently #9,042, which is down a bit from last week).

    Of course, I do 99.8% of my browsing in Firefox, and a substantial proportion of my readers also use Firefox, so… Shrug.

  • none

    Comscore is a joke, the only ones running comscores software are users that are to dumb to uninstall spyware. This is the main reason why slashdot is sooo under represented.

    1. Major american banks will not allow a user infected with marketscore access their sites. This is because marketscore/comscore breaks encyption and reads all your passwords, creditcard info etc.

    2. Marketscore is blocked by major universities across the united states. Read up on marketscore/comscore. Searching on google will quickly show you how to block and detect.

    3. Marketscores infection rate that i’ve seen is about 1 in 5000 visitors. This number jumps wildly from industry to industry.

    4. As for alexa is it just as bogus, you need 29 alexa surfers a day for a rank of 2000, you need 200/day to reach the top 400.

    In short the numbers they come up with and use are completely bogus.

  • Pingback: BuzzMachine » Blog Archive » Measure this

  • http://www.safetygearhq.com Bryan R.

    Speaking of Alexa’s scoring system. It is far from accurate. However I have noticed that when my site reached the 100k mark (or thereabout), Alexa began crawling and updating more regularly. At this time I checked my server stats with Alexa’s stats, and they were oddly similar.

    So, in closing, if you have 200k+ Alexa ranking, you can forget about even being updated regularly and this service will be useless for you. If you have 150k- ranking, then you should see a decent trend that follows your actual traffic.

    This is my take. Someone else may have seen different results.

  • Pingback: TechCrunch » Web 2.0 This Week (August 7-13)

  • Pingback: Top 25 populairste blogs - Frankwatching