A taxonomy of transparency

I’m writing the section of Public Parts on truly public government — transparency leading to collaboration. I am trying to come up with a simple taxonomy of transparency, a list of what should be open by default. Help me with my definition and list of buckets:

* * *

The first step to public government is transparency. My definition: opening up the information and the actions of government at every level by default in a way that enables any citizen to take, analyze, and use that data, extracting or adding value to it and overseeing the actions of those who act in our name, with our money. That data should include:
• Our laws and regulations—as they are being considered and after they are enacted, showing who did what to each along the way.
• Government budgets and spending, including information on who is paid.
• Government’s actions. I want to see crimes, complaints, actions, conferences and other events, even useful correspondence.
• Government information of every sort. The New Republic says making weather data public “produces more than $800 million in economic value.” Global positioning data enabled the creation of now-indispensible smart phones and navigation systems. Agricultural data saves and makes money on farms. If the government knows it, we should know it.
In What Would Google Do? I joked—well, half-joked—that the Freedom of Information Act should be repealed and turned inside out so that we no longer have to ask government to open up our information; government must ask our permission to keep it from us—especially now that technology gives us the tools to make use of it. There are a legitimate reasons not to release data: because it reveals personally identifiable information about citizens (e.g., your tax bill—though, again, that’s published in Scandinavia) or it compromises security or criminal investigations. Other than that, our information is ours and so we need access to it. In our developing information economy, this data has real and growing value. So cough it up.

* * *

Of course, the discussion is all the more timely thanks to Wikileaks’ latest deluge of once-secret data. The further question is how anything can be held in secret and what the appropriate line is for secrets. In the midst of the last Wikileaks disclosure, I suggested that the only cure for leaks is transparency: when the public trusts that a secret is secret for good reason, then revealing it is more clearly a violation of norms and the common good. But when most government actions and information are held from us, then exposing them is more just but will bring the collateral damage of also exposing things that should properly be held in secret. Whom do we trust with that judgment? The government? Wikileaks? The Guardian and Times? That’s what is being wrestled down right now.

In any case, I’d appreciate any help with the organization of thinking around what’s properly public and not. Thanks.

  • Jeff –
    A few thoughts.
    1. When dealing with government transparency you have the issue of standardization. That is to say – if a government can be fully “transparent” while releasing data on PDFs — a completely unworkable format for allowing citizens to remix that data. I would argue that this does not embody the spirit of transparency (citizen empowerment) that you are getting at. Thus, the first part of the transparency bargain is to release data in an easily parsable, remixable format. It is often in the remixing of data that insights are revelations are found.
    2. I like what you have above – At the risk of oversimplifying:
    Laws and regulations – A government creates laws to improve the lives of its citizens – therefore all laws and regulatory codes will be made freely available to the citizenry
    Spending – all pubic spending is made publicly available and all beneficiaries of public spending are clearly identified
    Taxpayer generated Data: Essentially the taxpayers fund all government data creation – hence they own it. This covers a wide variety of items you mention above – GPS, etc.

    • Very helpful, thanks. I like your elegant taxonomy. I was afraid I was leaving things out but I think your umbrellas cover nicely. And, yes, the very next paragraph quotes this Princeton proposal that govt agencies should not build web sites but out out data sets (and then build web sites along with citizens atop them): Yale Journal of Law & Technology, Vol. 11, p. 160, 2009 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1138083##

  • Hi Jeff
    The Government of Spain has changed several of his ministers because he wants to “communicate”, “explain” better to citizens what they do and why they do it. These are the reasons that he gave himself Zapatero executive.

    It is an attempt of this Government to improve its image in the polls a year and a half of elections.

    Thank you very much
    Ignacio Jaen

  • There are two goals of transparency that can sometimes find themselves at odds with one another:

    Transparency — to make things visible
    Transparency — to make things less opaque

    One technique for making things opaque (difficult to understand) is to overwhelm us with information. The Internal Revenue Service income tax code, for example, is famously visible (each item is published) but is famously opaque (impenetrable) at the same time.

    By the way, I have always found the term “transparent” to be amusingly contradictory. On the one hand, it means making things visible by removing a covering; on the other hand, it means making things invisible, as a pane of glass is invisible when it is completely transparent.

    My dictionary, for example, defines “transparent” computer software as “operating in such a way as to not be perceived by users.” At the same time, in fiction it defines “transparent” as manifest and obvious, as in a “story with a transparent plot.”

  • John Rigdon


    Tim O’Reilly’s Gov2.0 project has many of the same goals that you articulate here. That project is quite advanced. I hope you are aware of the efforts Tim is making in this regard.

    I look forward to your new book.

  • nitpicker

    The source of any money that enters the political argument should be available to all who vote. This includes the donors to organizations that run political ads.

  • What about actions by non-governmental political organizations? When a lobby or PAC has as much (or more) influence over policy as politicians, shouldn’t they be required to be as open?

    Also, I think it’s important to emphasize that the data must be easy to get and manipulate. If it is made available in a format that is cumbersome and hard to use, it might as well not be available. It’s like telling somebody they have the freedom to say whatever they want to say, but they can only say it in the middle of the ocean where nobody can hear them.

    • Andy Freeman

      > When a lobby or PAC has as much (or more) influence over policy as politicians, shouldn’t they be required to be as open?

      In this world, that’s like worrying about unicorns.

      It turns out that PACs and lobbies can’t have as much power as politicians.

      Remember, politicans cast votes that create (or not) laws. PACs and lobbies don’t.

  • Nice idea, I like the concept of a taxonomy of transparency.

    As well as considering the data itself, another way of looking at it might be to classify the reasons it is being released. From what I’ve seen of the debate in the UK, there seem to be three main reasons: (1) To unleash ‘armchair auditors’ to improve efficiency; (2) To improve transparency and trust in the political process; and (3) Capture the economic value in the data and promote growth.

    But looking at what is being done with the data, what seems to be happening in practice is a set of location-based services, data visualisers and comparison apps.

    This seems to go some way to meeting goal 3, but the other two seem somewhat in doubt at the moment (Govspark not withstanding). And perhaps they can’t be met without linking the open data movement in with freedom of information, as you mention, because the context of the decisions to spend money, etc, is needed if the value of it is to be usefully assessed.

    As I posted here, it would be a shame if the open data movement ends up taking flak for not delivering on its promises, when what is expected of it now may not be what it can actually deliver.

  • Karen

    Absolutely everything should be public, eventually. No one can know what’s of value to the public unless the public can see everything they’ve got and decide for themselves – important info will always be overlooked by people in charge who don’t necessarily understand its value or concealed by self-serving bureaucrats and politicians who will never be held accountable if there’s any way to keep something secret. So in issues of national security where lives are at stake, the government should be allowed to delay release of the info for say 20 years or until the threat is over, whichever comes first. And there should be an independent court to decide what info gets delayed – that is not a decision that politicians or bureaucrats should be allowed to make. Personally identifiable information (like my tax returns) should be held for say 100 years after my death so that it’s only real value is to historians or other researchers.
    Ultimately, if we’re to have truly accountable government we have to know absolutely everything, sooner or later.

  • Brad

    I can highly recommend looking at the Australian Gov 2.0 Taskforce report and the government response from their Department of Finance. It fully endorses a pro-disclosure culture with the default position that government data and publications should be released under CC-BY to enable widespread reuse.

    Joshua-Michéle Ross also makes a good point. Open data should be in open formats. Preferably XML-based for easy machine readability.

  • Sounds like you’re talking about Gov 2.0 to me, professor. Much of what you articulated also falls under the mantle of open government.

    I’ve been reporting on this topic all year. The pieces I’ve written at the Huffington Post on open government on the civic surplus and participatory platforms may also be of use, with respect to the thinking of federal government’s current technology leaders.

    If you haven’t talked to Ellen Miller at the Sunlight Foundation and Tom Lee at Sunlight Labs, I highly recommend both for an interview on transparency.

    Carl Malamud will also be invaluable, particularly with respect to Law.gov, which gibes with your first bullet. WIth respect to transparency, the simply open government data principles articulated at his site should be of particular relevance:

  • Everything that doesn’t breach personal privacy laws, commercial confidentiality or national security should be publicly available in machine readable formats by default.

    Realistically there’s a much shorter list of what should not be public.

    The challenge is re-engineering public sector culture and systems to facilitate transparency.

  • I believe some of it is already there at http://www.data.gov

    The trouble can be that even when information is out there, too few of us will take the time to review it, understand it and act upon it.

  • Pingback: Listen to 'A taxonomy of transparency' narrated by professionals, from 'BuzzMachine' - Hear a Blog()

  • I work in records management for a (US) federal agency. My program in our relatively small field office generates probably millions of pages of (internal) content annually. We grab a few hundred thousand of those pages, designate them as records, manage them in a database application. That application is currently internal only, but we make small pieces of it available to the public on the internet.

    We are in the middle of an effort to increase the portion of the pieces publicly internet available to perhaps 10% of the current electronic collection. This effort is technically complex and moderately expensive. The agency’s, program’s and office’s budget is continually under pressure to reduce expenses. Neither the Office of Management Budget (OMB) nor the agency have any established goals or budget designations for internet publishing.

    The discussion about government transparency should weigh cost, timing and scope. I don’t have an estimate of what it would cost to make “everything” from my agency available on the internet, but the cost would be orders of magnitude more than the existing web publishing budget. Timing particularly relates to the mountains of legacy paper records the government owns. Are they all worth digitizing and publishing? To date Congress hasn’t seen fit to pay for that. Perhaps it makes more sense to move towards day forward web publishing than to work on the old stuff. Scope is the fundamental question. Is it really worth publicizing every small purchase, every draft document, every vehicle mileage log, every .gov email? If that is the direction of public policy, the resources for it will have to come from somewhere. Somewhere will probably be existing programs or much less likely new revenue.

    • Dave,
      OK. Now imagine another scenario. Imagine that you used, say, Google Docs for everything you do. (Nevermind the brand; the point is the method). You could do most everything you need with that: text, spreadsheets, presentations, forms. (Indeed, wouldn’t it be great if I could fill out government forms this way?!!) Now that wouldn’t include other things, I know: data bases and such. But it would be a lot. So now imagine you have a simple toggle on each doc: public or not, publish or not. OK, but then you end up with a huge mishmosh of docs, I know. But they can be searchable by keywords — no worse than the mishmash of the web, eh? They can be tagged by topic and interest group: official tags, even. They can also be tagged with department and source. It wouldn’t be a neat, clean filing cabinet. But neither is life. At least it would be available, searchable, public, ours. Done that way, I’ll just bet it could *reduce* costs by getting rid of expensive desktop software. Make the cloud public.

      • Great points all around. I administer a local municipality’s Sharepoint Intranet, Extranet & Internet system.

        I’ll simply point out that it would seem to me that if Google, with their resources, can build Google Apps (+docs), the Federal government should also be able to build/borrow and scale a similar system. Instead, what we have are thousands of government agencies paying our tax dollar through the nose for their own individual – and rarely open – web-based CMS systems.

        Sure would like to see politicians talking about using technology on HUGE levels to find these cross-agency efficiencies as well as build in the openness that the new IT market demands.

        Santa Rosa, CA

    • The point about cost is an extremely valid point. Technology usually makes systems (corporate or government) faster and more efficient. This does not always translate to savings, though. I would contend that the money for digitizing old documents should come from the paper budgets of the respective agencies following their complete changeover to paperless systems.

      • John,
        That was true when huge government systems were spec’ed. But when open standards are used without capital expenditure, it can greatly reduce costs. Problem is that the attitudes are way behind. I’m visiting a government office next Wednesday and have to leave to make the This Week in Google podcast because the govt IT won’t allow access to Skype! Oy, I say.

  • John (UK)

    Jeff and all,

    Some interesting points being raised and I’m a little surprised not to see any mention of Sunlight Foundation who, in my limited understanding, take transparency to another level: is the information being provided useful i.e. loads of data can be meaningless.

    Perhaps this has already been covered in another earlier blog or maybe O’Reily’s Gov2.0 covers this?

  • Hi Jeff
    I’m a Yank living in New Zealand (NZ) and I believe NZ’s most redeeming quality is ‘transparency.’ I was born and have lived in the USA for about 25 years and I have also had the pleasure of living abroad for almost 20 years. Living in the country (NZ) that is ranked #1 (http://www.transparency.org/publications/annual_report) with regard to the lowest levels of corrupt enables me to see your remarks differently.

    The USA has a level of sophisticated corruption (ranked 19th least corrupt) as does Japan (rated 17th) where I lived for 3 three years. My experience of living in these three countries supports the afroementioned research; The veracity of the report is reinforced, particularly if you have had a chance to spend time in India ranked 79th with constant chirps of “Baksheesh” ringing throughout the city and China (84).

    My point is that transparency is not about technology, but rather an inveterate behaviour and value deeply interwoven through culture. When the ethos of a society is transparency then and only then can a technology support such behaviour, not the other way around.

    I believe you will agree that the internet is an enabler and a great equaliser when it comes to openness and transparency, but it is only an enabler if the values of society embrace such norms. Something China is unwilling to do at this stage. You can make all the technology comply with ‘transparent standards’, but unless societal behaviour supports this…you may be advocating an act of futility! And from my experience there is still some distance to cover.

    Thanks for your ideas and I enjoy listening to you and the crew on TWG.