Nothing ever stays the same: week to week anomalies in R2K polls

Research2000 published political tracking polls, where they ask the same question, week after week. The results of such polls can be compared over time:

From this, we can look at the change from week to week, and then count how many 1% changes occurred, 2% changes, etc:

What is strange about this data? Look at the sharp dip at 0% change. Let’s compare this to the changes in a similar Gallup poll:

Ok, so it’s weird, but could it come from actual changes in opinions of the populace? What if there are weekly oscillations such that the public opinion never remains the same week to week?

Well, even if that were the case, this data would be very strange. Because there are LOTS of +/- 1% changes. 1% is very close to 0%, and with a margin of error of 2%, a 1 point change in public opinion will often show up as a 0 point change in the poll.

To illustrate this, let’s pretend the underlying opinion actually did have weekly changes of only +/- 1%, and then look at what a weekly poll of that opinion might look like:

When we measure this through polling, we will only be asking a randomly chosen subset of the population, and therefore, there won’t be the precise changes you see in the underlying population opinion. To Illustrate this further, let’s look at the histograms of weekly change for the underlying opinion and the simulated polls:

Notice how despite the fact that the underlying opinion has no 0 point changes, the poll of that opinion has lots. What are the odds of it having as few as R2K’s polls have had in the last 74 weeks?

One in 10000000000000000

NOTE: Please see our recent report for more detail, or come back here for future technical write ups. The figures in this posts are just illustrations, check out our data section for the real data

Posted in Uncategorized | 1 Comment

Measurement vs. Estimation

This is a short post because I’m on field trip in rural texas, another post with graph and explanation will be up soon.

In this statement, Research 2000 head Del Ali seems to state that he manually altered his polling data, though it’s hard to tell from the muddled language:

Yes we weight heavily and I will, using the margin of error adjust the top line and when adjusted under my discretion as both a pollster and social scientist, therefore all sub groups must be adjusted as well.

He also seems to say that this is common among pollsters:

I challenge anyone to then look at comparable data from other firms, not one or two but many others.

This is actually a very good point. In political polling especially, there seems to be an almost deliberate murkiness about methods. I think this is because many polls may not be actual data reported directly from polls, but rather estimations computed using outside information and only partially based on the polling sample.

While there is nothing wrong with estimation as opposed to raw sampling, it is important that a clear distinction be made between the two.

Posted in Uncategorized | 1 Comment

Which polls to trust?

We’ve found at least two pollsters with very strange data (Strategic Vision and Research2000) but does that mean you should trust the polling organizations with data that appears normal?

The thing is, if one were to fake data (not saying anyone did), it wouldn’t be hard to do it so that it would be undetectable to our methods of analysis. People will realize this, and it would be a shame if the only lesson to come of this is how to fake data properly

Ultimately, we need the media publishing the reports to show complete transparency, to demand as much of the detailed data as possible, and to hold their pollsters to high standards.

Actually, this isn’t just for polling. All documents and media sources are possible to fabricate, so it’s important that it be as open as possible. No one will notice the flaws if it’s kept in the dark.

Posted in Uncategorized | 2 Comments

DailyKos Story

Detailed report:
Posted today!

Kos’ remarks:
Posted right after!

I’ll be putting up similar posts to the one below, to help explain long and boring things with pictures.

Posted in Uncategorized | 3 Comments

Men and women having strange relationships

Pollsters keep track of the gender of the people they call. This means that any poll can be broken down into two polls with entirely separate sets of respondents. The private polling firm Research2000 reported the gender breakdowns for all 795 [Note: This number was wrong earlier] of their weekly tracking poll questions done for dailykos.



These two separate smaller polls may look similar if men and women have similar opinions, but there is no overlap in respondents, so the precise % numbers are completely unrelated.

This means that if an odd-number percentage of men are “FAV”,  it does not have any bearing on whether an odd or even % of women say “FAV”.

That said, we can look at all the questions that Research2000 asked, and count how often the men’s results and the women’s matched in being even or odd.  For each question, there are four possible combinations:

Because the results come independently, each of these combinations is equally likely.  So for the 795 weekly questions,  we should have roughly 199 of each. Here are the actual tallies:

(Among “FAV” responses in questions with 3 possible responses asked in state polls only.)

How likely is this to happen by chance?

Well, this is roughly the equivalent of flipping two different coins 795 times, and only getting different results twice. It should happen by chance approximately:

One time in
10000000000000000000000000000000000000000
00000000000000000000000000000000000000000
00000000000000000000000000000000000000000
00000000000000000000000000000000000000000
00000000000000000000000000000000000000000
00000000000000000000000000000

Posted in Uncategorized | 8 Comments

Broken meters

News agencies are constantly reporting on polls and statistics. Often the same survey questions are asked week after week, and reporters give them back to us as if they were the pulse and pressure of the nation.

But how are these numbers generated? Who is doing the surveys, and how accurate are they?

We hope to shed some light on that issue. Just like blood pressure measurements, some patterns of numbers are extremely unlikely, and may indicate that there is something amiss with the meter rather than the thing being measured.

Posted in Uncategorized | 2 Comments