Credibility is not binary

I’ve seen a couple of efforts lately to help determine who’s credible online and though I understand the need and the motive, these attempts are fundamentally flawed and perhaps even more damaging than they are helpful.

The latest is Newscred and Techcrunch describes it today. I used a Techcrunch beta invite to poke around.

It’s very simple — though that’s the problem; credibility isn’t so simple. They list articles and you get to “credit” or “discredit” them. These scores are, in turn, compiled for writers and publications.

The first and most obvious problem, which TechCrunch points out, is that this is bait for grudges. Fox from one side, the Times from the other will get discredited by their detractors all day long. One man’s bias is often the other man’s truth.

The second and more fundamental problem is that there’s no basis to decide credibility. Does one error ruin an article’s credibility? How many discredits does it take to ruin a reporter’s or a publication’s? And then what does that mean? That they lied? That you don’t believe them? That you don’t like them? That they make mistakes? That they don’t report enough? That they use anonymous sources? That they relied on bad sources? That they wrote it badly? That they weren’t transparent?

And who’s doing the judging? Are they credible? Who’s judging the judges, then?

Over the years, I’ve heard of various attempts to determine credibility or bias algorithmically, in an effort to take out this human bias in the process of finding bias, but that’s just an engineer’s wet dream. Again, the problem is definition (not to mention technical limitations of analyzing text and ideas).

Newstrust has tried to do this in a subtler way, with star ratings and comment, but it faces the same issues: Who’s doing the rating? On what basis?

I think these folks are attacking the problem from the wrong perspective. They’re trying to play whack-a-mole with credibility and identify all the bad stuff — just as news people, long accustomed to packaging the world in a pretty box with a bow on top, keep wanting to kill every bad comment on their sites. They’ll fail. Life insists on being messy. The task of identifying the bad stuff is so large — there is, indeed, too much junk — that these folks try to scale their effort with simplicity or technology. Won’t work. They’ll never find all the bad stuff. Ultimately, this can be dangerous because good people who do good work can easily be besmirched by bad judges with grudges.

Instead, I think it would be far more useful to concentrate on finding the good stuff. That is the real challenge in the new architecture of news and media, in the ecosystem of distribution and aggregation. When all the articles on a given topic are brought together by Daylife (where, disclosure, I am a partner) or Google News the need and the true service is to find the best articles because that’s what we want to spend our time on. (A restaurant guide with only bad reviews doesn’t help me eat.)

We also need to find ways to surface original reporting so we can support that reporting with our attention (and with traffic and ads). This is why I believe that there should be an ethic in professional journalism, as there is in blogs, to link to prior work and sources. All roads should link back to the original reporting.

There is still clearly bias in this approach of finding the best. Many will recommend Paul Krugman, many won’t trust that recommendation. Who’s doing the recommending still matters (and so it would be very helpful to have transparency among them). But by highlighting the good rather than trying to expunge the bad, we would try to support good journalism wherever it is done — msm or blog. And that’s really the point, isn’t it?

On top of that, every news site should have a means for people to help correct errors — that’s as simple as adding comments (though doing so does add a cost to police them). Correcting errors makes one more credible; that, too, is an ethic of blogs. And that, too, will improve the journalism, just as you improve mine in comments here. At the end of the day, there’ll always be disagreements, though. Look at the post below about airlines; there’s plenty of argument there. Is that really about credibility? No. It’s about conversation.

  • http://robertdfeinman.com/society robertdfeinman

    Victor Navasky summed it up on Bill Moyer’s show two weeks ago.

    ” A lot of years thinking about it. The format of journalism is that you quote someone on one side, and then you quote someone on the other, and you have to pick experts. And the theory being that if you get two people who, as we found out in The Experts Speak, two experts who are wrong, that somehow you’re gonna get the truth out of that. In the case of why did The Times hire Bill Kristol, however, there’s a case where I think that it’s a principle of the op-ed page. If you have people on one side — Bob Herbert or someone like that — and he happens to be right, you need someone who happens to be wrong on the other side. So, it’s — that’s one of the principles of balance that happens in the news business. And–”

    Truth now gets “balanced” by lies.

  • http://www.newscred.com Shafqat

    Great points Jeff. One of the goals of NewsCred has always been to help reward great journalism. We want to help our readers ‘find the good stuff.’ Perhaps we can make it more clear on the site, butt we’re not interested in pointing fingers, accusing or being the negative food reviewers as in your example. I think we all agree that finding the signal in all this noise is the end goal. By crediting and discrediting, we simply hope to be able to filter. But the explicit votes are only one aspect of our site. We use implicit inputs like a journalist or source’s past record to also assist the user in finding high quality content. We also want to help build online track records for journalists, something that we think is beneficial to both readers and journalists.

    We’re only three weeks into our alpha, so we treasure good feedback. While we are more optimistic about the ability of our community to determine quality journalism, we’ll certainly take all your comments into mind as we set sail on what is surely a long, difficult but ultimately rewarding journey. Let’s keep in touch!

    Shafqat (cofounder of NewsCred)

  • http://justenoughtechnology.typepad.com Michael fitzGerald

    Credibility ultimately depends on getting the facts right. It’s hard enough for journalists to do that. Facts are binary “credibility” is another word for “likeability”

  • http://www.newscred.com Shafqat

    By the way, just following up on the ‘celebrate the good journalism’ theme is this post that I wrote a couple weeks ago reiterating the reasons behind NewsCred: http://blog.newscred.com/?p=66

  • http://theleftovers.wordpress.com/ Boxcar Children

    You bring up interesting thoughts. Whenever I see a grammatical error in a news source, I automatically begin to wonder about the credibility of the source, regardless of where it is. And because people of the Millinial generation or Generation Y have grown up with the assumption that most internet sources aren’t very credible, will internet editors become more prized positions in companies? It seems likely that companies will begin to invest more in their internet editors and writers as marketers of their credibility in the future, if they aren’t already.

  • http://fasterfuture.blogspot.com david cushman

    Jeff, I only find credible that which people I trust find credible.
    Any form of lowest common denominator construction of this misses the point.

  • http://www.google.com/notebook/public/09184762010473690803/BDQZVQwoQqIT66v8i flingcom

    We’re only three weeks into our alpha, so we treasure good feedback. While we are more optimistic about the ability of our community to determine quality journalism, we’ll certainly take all your comments into mind as we set sail on what is surely a long, difficult but ultimately rewarding journey. Let’s keep in touch!

  • http://www.newscred.com Shafqat

    David – thats a good point. We’re thinking about allowing our users to personalize NewsCred so that their particular version is based on credibility ratings and profile of users they chose to follow. So you can set up a group of people you trust, and then we’ll server you news based on recommendations and ratings of that trusted group. Does that make sense?

  • http://www.newstrust.net Fabrice Florin

    Hey Jeff,

    Thanks for including NewsTrust in your arguments about online credibility.
    (Disclosure: I am NewsTrust’s Executive Director)

    As much as we appreciate the coverage, I would encourage you to take a closer look at what’s actually going on at NewsTrust before calling our methodology ‘fundamentally flawed.’

    May I suggest that we spend some time on the phone next week, so I could give you an update on our service? It’s been a couple years since you originally checked the site and last reviewed on NewsTrust, and we have made some significant progress since, that addresses many of your concerns.

    For example, our stated nonprofit mission at NewsTrust is to ‘help people find good journalism online.’ I think we do a pretty good job of it, if you’ll compare our feeds to other aggregators. Not only that, but we also help citizens develop their own news literacy skills and become more discriminating news consumers in the process. Those are all essential ingredients in building a culture of trust, based on facts and core principles of journalism, not just opinion or popularity.

    We also go to great lengths to track the reputation of our own reviewers, both through member ratings and staff validation, with criteria ranging from expertise to transparency and fairness. We then weigh each reviewer’s ratings on our site based on their overall member level, which gives our most trusted members more influence on the final ratings.

    So while we don’t claim to have all the answers, we are demonstrating clear progress with a reasonable and practical solution that deserves a bit more careful attention than you were able to devote above. As a result, this blog post seems rather superficial and largely based on the author’s speculations — with no factual evidence, authoritative sources or independent perspectives to back up its arguments. Who’s calling the kettle black?

    I can understand your concern about the risk of oversimplifying our quantitative evaluations of this qualitative process, but I encourage you to take a closer look at the merits of the hybrid approach we practice, which combines the wisdom of the ‘vetted’ crowd with that of a few good editors and some pretty sophisticated computer algorithms. The resulting news filter we’ve evolved through this three-way interaction is much more nuanced than the ‘binary’ adjective implies.

    As a society, we need new ways to sift through the ‘information tsunami’ that hits us every day to find quality news we can trust. But solutions that rely only on computer-assisted filtering, like DayLife or Google News (both of whom we have the highest respect for), fall short at some point, because we haven’t yet figured out how to teach computers the difference between good journalism and misinformation. And other solutions that solely rely on thumbs-up/thumbs-down popularity contests like Digg or Reddit, also fall short because of their lack of discipline and shared values.

    So we think that a combination of professionals, amateurs and computers, guided by core principles of journalism and a nuanced rating system, offers the best of all these worlds. A conversational approach to sifting through the news is wonderful, but how’s a computer supposed to make sense of all these words? Computers like numbers and binary data, so inevitably, all these qualitative judgments will have to be converted into numbers before we can properly sort them out.

    You can find out more about NewsTrust on our site and on our blog — and if you’d interested in a full overview, check the video of our presentation at Google a couple weeks ago: http://blog.newstrust.net/2008/05/newstrust-at-th.html

    Meanwhile, I look forward to comparing notes with our friends at NewsCred to see how we can help each other address these important challenges. And we’re already working with DayLife to add value to our service by combining some of their meta-data with ours. Ultimately, I believe the scope of this problem will require an alliance of compatible and thoughtful initiatives working closely together to bring about a solution that works for the public.

  • Pingback: NewsCred Blog » Blog Archive » What We Are and What We Are Not

  • Pingback: Andreas Cancigous Blog » Blog Archive » jennifer gareis sex videos

  • Pingback: blog kristal » Blog Archive » jiggling ass free clips

  • Pingback: refatas » Blog Archive » her suking first big cook

  • Pingback: free nicole sheridan video clips » Blog Archive » adult sis sex games

  • Pingback: Hate, props, and hotness: restoring journalism’s credibility | Sixth W

  • Pingback: Digitale Notizen » Blog Archive » Glaubwürdige soziale News