The Citizens’ Survey: Open-source polling
: Question: Is it possible to create an open-source polling service that gets a more accurate picture of public opinion and also allows the public to not only answer questions but also to ask them?
Yes___ No___ Don’t know/care___
I think we could create a community-based polling service that answers many needs and deficiencies in the opinion industry today:
: Lots of questions that need to be asked aren’t getting asked: I have my own pet polling wishes: I’d like to see more questions that really determine whether we are a nation at red-v-blue war or whether that is the fantasy of the fringes and media. I’d like to see how many people think the Parents Television Council speaks for them. I know you have your own questions.
: There is what I sense to be a growing storm about bias in polls: One response to that is to have competing polls.
: Polling is innaccurate: Exit the exit polls. I have a theory they’re screwed up because people are gaming the system out of a hostility to polling not unlike the hostility to big media.
: Polls are affecting — and can thus skew — the public discourse and media coverage: Would politicians govern better and the press cover better if they had a truer sense of what their constituencies think?
: New businesses cannot afford market research: Imagine the businesses that could be created if entrepreneurs could sniff out new needs.
: Polling is too damned expensive.
So why shouldn’t the opinion industry find new competition in an open-source citizens’ effort?
I first started thinking about this as I wondered how to get my pet polling questions asked and answered. I wondered whether there could be a way to get established pollsters to charitably add questions to their surveys. But, of course, that’s utterly unworkable: politics, money, work, logistics, and self-interest all get in the way. But then it occurred to me that it would be possible to set up a system for the people to take over polling:
It’s the wikification, blogification, Craigsification, bittorrentification, linuxification of opinion.
The idea is simply that an open polling service allows anyone to answer questions (when they meet sampling requirements) and ask questions (with gating by the community). The requirements (I think):
: It must have scientific sampling: The system has to know the demographics of the nation, gather the demographics of the respondents, and create samples to accurately reflect the public or a slice of it (e.g., young people, women, parents, Democrats, FoxNews viewers…). That requires expertise.
: It must defeat gaming: This is the killer. The obvious fear is that Republicans will masquerade as Democrats, or vice versa, to fake the results (“95 percent of Democrats think Hillary is an alien”). I think the way to deal with this is to align the interests of giving answers and giving personal data: We create a methodology to define “conservative” not just by asking you whether you’re conservative (though we can do that) but primarily by your answers to a larger series of questions (taxes, federalism, etc.) and you won’t know which are questions aimed at determining your categorization or questions from polls to be published. If you want to get your conservative views into this statement of American public opinion, then you are disincented from lying about whether you want to raise taxes, for example.
Also, Wikipedia style, we let the collective define the questions that define the descriptive words: Does tolerance of a federal deficit make you conservative or liberal today? If the collective can’t agree on that, then deficit attitudes do not make up a definition of political ideology.
And, importantly, we do not allow respondents to pick the questions they will answer: You can’t come — USAToday poll-like — to flood the results. The system asks you what it wants to ask you based on its needs to fill certain samples and ask certain questions.
: It must attract a critical mass of respondents: Why would anyone bother answering questions here? I think they would if they knew they were participating in a poll that could influence discourse, government, and industry. I think most of us have enough ego to love to be asked what we think about anything (or is that just bloggers and pundits?). And I think the apparent randomness of the questions you’re asked — so long as they’re not too laborious — makes this fun
Also, if you want to ask questions, you have to answer them to get influence points to push your questions. Perhaps you can align yourself with groups trying to get questions asked and donate your influence points to that group so the group’s questions have a better chance of being selected.
: It must prioritize questions: It can’t ask every version of every question of everyone. So the system needs a means of first pooling similar questions. (Think Flicker: You tag your question “Democrats, Hillary Clinton” to see what questions have already been submitted.) Then the system needs a means for the community to decide which questions get asked (this is where your answer points come in) based on a calculation of system capacity (based on total available respondents in demographic groupings and the number of questions a respondent can be expected to answer) and community demand.
: It must edit questions: Asking polling questions is both a science and an art. The system needs to collect the wisdom of both the crowd and of experts in forming questions in a way that results are meaningful. I think this could be done in wikipedia style with experts — pollsters, statisticians — getting added juice. There is also a need for FAQs and even wizards to guide people through the best way to ask a particular kind of question.
: It must be transparent: Otherwise, there is no way to ferret out bias and opinion spam.
: It must be free: So to support the system, I suggest that companies and campaigns be allowed to use the system for a fee. And wouldn’t this be a heckuvan environment for advertising (you can reach soccer moms for sure).
Could this be a for-profit company? Perhaps. But I don’t think this will work unless the community believes it owns it. I think it needs to be public and transparent.
There is another important benefit that comes from doing this online with a large and ongoing poll of respondents:
The system brings context to polling. It can create panels of individuals and ask them the same questions over time to get a sense of shifting opinion. It can become self-correcting: It can find out whether likely Bush voters really voted for Bush so it can start to measure the likelihood of “likely.” It can also link to other data as a check on results (e.g., the poll says people don’t care about Michael Jackson news and Technorati links back that up among bloggers but the Tyndall Report finds lots of Jackson coverage).
I can’t build this. And for all I know — and I’m sure you’ll tell me — elements of this already exist somewhere. But I’d love to see some brilliant stats wonks bring it all together.
So what do you think?
LIke it___ Hate it___ Don’t know/care___
