Scaling fact-checking

Before Thanksgiving, CUNY’s Tow-Knight Center for Entrepreneurial Journalism convened a meeting of three dozen journalists, technologists, librarians, entrepreneurs, and academics to discuss ways to scale fact-checking.

The event was born out a conversation with Craig Newmark, who helped fund it. Improving trust in the press and battling disinformation are among the causes he supports. There are fact-checking enterprises already doing good work — most notably, and Politifact. Craig Silverman of Regret-the-Error fame, gave us a great presentation on the history and state of the art in fact checking.

But these find efforts and organizations can do only so much. And there are so many lies, distortions, and mistakes out there. So the question Craig and I discussed is how to scale fact-checking — and awareness about the need for it.

That led to the event. Here is my colleague Jeremy Caplan’s exhaustive Storify compiling tweets and more from the event. Craig’s takeaway is here. And here are my notes. After hearing the room, I came to see that facts face supply and demand issues.

* Supply of facts: We need more effort to get more government and business information made public in useful forms. There are organizations like The Sunlight Foundation, represented at the event, that are trying. But I believe we — especially journalists — should be campaigning to make government — as I argue in Public Parts — transparent by default and secret by necessity. More data made public is good for many reasons but one of them is simply increasing the supply of facts.

* Supply of disinformation: Jay Rosen argued that we are seeing a disturbing trend in “verification in reverse:” taking a fact and unmaking it, until people don’t believe it anymore. He cited the birthers and climate-change deniers as well as Mitt Romney’s much-fact-checked and debunked campaign commercial. He said there is a growing supply of “public untruths.” He argued: “Verification in reverse should be a beat… We have to start ranking public untruths by their seriousness and spread — we have to start IDing the ones that are out there and influencing public conversation, even though they’re already being fact-checking… We have to start acknowledging what’s going on with systematically distorting truth…”

* Demand for facts: Part of the challenge, the group said, is to increase the demand for fact-checking among journalists and the public — and maybe even politicians.

That leads to:

* Practices: The Washington Post and the Torrington Register Citizen began putting fact-check boxes on their stories. That, to me, is an incredibly simple way to open the opportunity for facts to be challenged and corrected and to make constant correction part of the process. What else can we do to bring fact-checking to the fore?


* Standards: Joaquin Alvarado, VP for digital innovation at American Public Media, threw out the challenge to begin standardizing how we store and present facts in media so we don’t have to waste effort and so there is an easy means to point the public to already verified information. This won’t be as simple as a spreadsheet; facts require explanation and examination. But Joaquin volunteered to get appropriate parties together to get a start on standardization.

* Tools. See Jeremy and Craig Silverman above for links to the neat tools some are creating, among them Truth Goggles, a project at the MIT Media Lab.

* Culture and education: CUNY might hold a next event on making facts fun. Sounds silly, I know, but many in the room believed that fact-checking needs to be made into a game. And Craig pointed out that some of the best fact-checking out there is done by Jon Stewart et al: truth as entertainment.

* Research: The New America Foundation is holding another event on fact-checking in December, concentrating on research about effectiveness of various methods: what works, what sticks? That is vital to make best use of the precious resources we have.

And finally, that leads to:

* Sustainability: Fact-checking is expensive. All the efforts above try to make it more efficient, by increasing the supply of facts, by getting more people involved, by creating tools, by adopting standards. This is where the Tow-Knight Center for Entrepreneurial Journalism gets involved: I believe that more transparency and more collaboration will help make for more efficient and sustainable journalism. We need to create and take advantage of existing platforms and then add journalistic value to them. We need to harness the care and energy communities already expend to share their own information. We need to help them do that.

More to come later. If I got anything wrong, correct me.

  • I think an important part of the system would be the automated repeal or updating of facts as new information becomes available.

    For example, say a nutritional study comes out on the health benefits of ‘Widgetjuice’. People start writing all the articles, recipes, and top 7 healthy ways to use Widgetjuice blog posts. Each (or at least some) use this new standardized way to reference the study as a data point to establish fact. Then 5 years later another study is done showing that Widgetjuice is in fact bad for you and that the original study was flawed. Maybe even some of the original data was good and Widgetjuice has some benefits but also previously unknown side effects. The system should allow this new information to surface in relation to the older articles that tried, in good faith, to use the system.

    Good luck, I for one can’t wait to be able to easily fact check what I am reading.

  • This is a fantastic conversation to be having. We put fact-check submission links on all stories on the CBC’s news, sports and entertainment sites more than five years ago and it got tremendously positive reaction from our audiences — not to mention helped us improve content (see example here I believe we were the first major site to do so. I’ve long wondered why more sites haven’t done the same – it seems like a simple and obvious step.

  • Subject: Fact Checking??

    Jeff, as an avid reader of several blogs, including yours, Doc Searls and Dave Winer’s, I have much more to say than Dave would allow in a comment so I have been posting to my blog as instructed.

    I have sent you copies via e-mail; however, your lack of response could have unknown causes. So, would you do me a simple favor and check it out and indicate your level of interest, which I will respect.

    Thank you, very much

    Doug Skoglund [email protected]

  • Michael K Pate

    I wish Jay Rosen would be more careful about throwing around “climate-change deniers.” Rosen should be more concerned about statements like the one below.

    “Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. – Dr. Phil Jones, UAE,

    Last time I checked, reproducible results were apart of the Scientific Method.

  • Marc Danziger

    Jeff, I have mixed feelings about this, which are neatly highlighted by your approvingly echoing Rosen on “AGW “denialists”. The problem is that once the ‘facts’ we’re discussing get past the did-he or didn’t-he stage, you move into a realm of assertion at some level. What you’re proposing is that respectable people (as determined – how?) – ought to do the asserting and the rest of us should stand in line. I don;t think you really believe that.

    But as someone with a scientific background and a lot of comfort with math, the AGW case has always been a very sketchy one – and new data and new information doesn’t carry it across the line to ‘fact’. Scale that to touchy and value-laden issues like economics or politics and I don’t see happy faces at the end of it.


  • Andy Freeman

    > The problem is that once the ‘facts’ we’re discussing get past the did-he or didn’t-he stage, you move into a realm of assertion at some level.

    He-said/she-said is pretty much as far as journalists can go because they don’t have domain expertise.

    However, a better “he-said/she-said” system would probably be an improvement. Instead of just baldly stating “facts”, providing provenance would be a big improvement.

    Of course, some of the sources might not like this, because then they’d get blamed when things turn out differently than they predicted. This might be a good thing for the rest of us. (For example, when using Barney Frank as an authority on mortgages, it’s surely relevant that BF said in 2007 that Fannie and Freddie were beyond reproach.)

  • Jeff,

    This was an interesting and important event. Missing from the conversation were researchers, news librarians, archivists and others who safeguard editorial integrity in the news every day, though I’m glad some of us could participate via Twitter.

    These journalists do this professionally, and are critical in maintaining accuracy.

    It’s great that the tools are in everyone’s hands now, but I think it is naive to think that there’s this highly motivated crowd, smartphone at the ready, eager to check all facts and receive online trophies and badges. Also let us remind ourselves that the Internet, news websites and commercial aggregators exist because there are archivists, news librarians and other professionals enhancing, classifying and transmitting content to these(often while doing a host of other things, including checking facts and creating content).

    There are several hundred journalists of this type working in this business, in broadcast, radio, newspaper journalism, and other emerging news forms, as well as Jon Stewart show as mentioned. Trust and accuracy is clearly important to these entities, as it should be if covering the news to contribute to an informed citizenry is your daily mission. If they are finding it too costly and time consuming, hope they still have some citizens and readers who motivated enough to safeguard the facts for them.

  • Andy Freeman’s suggestion of providing provenance is an excellent suggestion. We need a transparent process so readers can follow the logic of journalistic fact-checking. Of course, this would most effectively work online where there’s no space constraints and sources are linked for review at the reader’s convenience.

    As for Jay Rosen’s absolutist proclamation on climate science, it’s unclear how much provenance he has on the topic. But researcher Roger Pielke Jr., no “denialist” writes that it’s a bit more complicated:

    “I have seen from the inside many efforts by a small set of prominent climate scientists to bully and suppress — behaviors which continue even after the release of the UEA emails. Such behavior is seemingly emboldened by the protective shield that many in the media hold up to protect climate scientists from criticism, no matter how legitimate.”

  • John Cunningham

    Jay Rosen, and presumably you, consider “climate deniers” to be propagators of falsehood. He and you are deluded. read on the new release of emails among the leading AGW proponents. they proclaim the absolute certainty of the AGW case publicly, while telling each other privately that their models do not work, and that there is no proof of unprecedented warming in the past 150 years. how about doing a little independent research rather than toeing the Lefty line?

  • Andy Freeman

    suggests that journalists are playinig games with “fact checking”.

    I especially like the argument that “makes” means salary as I just got yet another memo from my employer touting the value of benefits, which doesn’t, for some reason, include a defined benefit package, let alone one guaranteed by the US taxpayer.

  • I think Jay Rosen’s points are well founded. The corporate / political deception machine is so pervasive that I see the dissemination of disinformation filter down even to social media sites. Go to reddit and look in the comment sections on political articles. In my view, persons with an agenda to distort facts are at work there. We all know this is (and has been) the case with the MSM but I am alarmed to see the practice having moved to alternative new sources on the web.

  • Pingback: Journalism via jokes « BuzzMachine()

  • Pingback: Journalism via jokes « BuzzMachine |

  • Anna Haynes

    > The Washington Post and the Torrington Register Citizen began putting fact-check boxes on their stories.

    Correction: While the Register Citizen has a prominent fact-check box, WaPo just has a gray fine-print link down among the misc page links ( underneath “work for us”, “community relations” etc., called “Corrections/Suggestions”) that I didn’t see when I first went looking for it. If others also find it hard to see, how about suggesting to WaPo that they instead use the Report an Error alliance icon, & put it up next to the “social media” buttons? (which are easy to find, right under the story)

    p.s. suggestion to Jeff: put the Report an Error Alliance icon (with a mailto behind it) on your blogposts…

  • Pingback: Journalism via jokes | Fast Fails | The Best Fail Channels()

  • Pingback: The Morning Lowdown 11-29-11 — paidContent()