Hi David:

I think the answer would have to do with the specific questions themselves. But it's probably more difficult to design questions that will produce accurate results right before an election, especially one like this one.

Best wishes,
Mason

The folks at 538 also caution their readers not to base things on the results from any one pollster, but rather to look at the combined results of a multitude of polls.

Polling by its very nature has an unavoidable degree of error - if you are taking a sample of 1000 people and trying to make a prediction about how 10 million people will vote on something...if you know what you’re doing you can get pretty close, but if you nail it exactly you simply got lucky.

538 also rates the various polling organizations based on their historical accuracy; if you are really into diving into the details about polling, you are probably quite familiar with their site already...if not, definitely worth checking out.

Party identification is not something intrinsic or immutable or a necessarily known quantity in the sample such that it can be used as a gauge for whether or not a given sample is representative. It can and does change relatively frequently for people, certainly compared to other demographic information. Race does not. Gender only very rarely. Age always changes obviously, but the true distribution of ages is pretty well known. Likewise with incomes. That cannot be said for party ID.

Put more simply, surely you can't adjust for sampling error in a variable which is by far the most accurate predictor of the variable the poll is designed to gauge?

But that’s the point. A sample being D+7 or whatever doesn’t tell you there’s anything wrong.

I’m unimpressed with appeal to (your own) authority since current pollsters don’t typically weight by party ID for the reasons I noted above.

Hi dragons:

I agree that 538 is a good site with lots of good information.

Best wishes,
Mason

Hi Wookie:

I think you miss the point. Everything you say may be correct. However, I have heard several times that certain polls are over counting Democrats in their sample, and I was just addressing that.

Best wishes,
mason

How does one know in advance that, e.g. too many Republicans have been counted?

Easy. Say I surveyed 100 people and found that 55 of them have ever worn a bra. I know a priori that men don't ever wear bras, so I assume that 55% of the people I surveyed were women, and adjust my results for the sample bias to extrapolate that 50% of the population have ever worn a bra.

Sounds like this adjustment is using similar logic.

Well, I don't know for sure what a particular pollster might be doing. But the percentage of Republicans among registered voters in a county/state I believe is public information.

So, for instance, if it's known that, let's say, in a certain area X percent of the voters are members of Party A, but the sample has Y percent of Party A voters, the results can then be adjusted by first multiplying by X and then dividing by Y.

Now does a particular poll do this in coming up with their estimates, there's no way I would know.

Best wishes,
Mason

Hi d2_e4:

I think this is correct. If you're a pollster and you see that there is something wrong in your raw data, you'll try to make an adjustment.

Best wishes,
Mason

Hi, thank you for doing this

How many people would need to be polled to have a reasonably accurate idea of who is going to win? lets say 99% confidence interval

Hi NewAcc:

First, you're welcome. However, please remember that my answers may not all be correct.

Second, you're making a mistake that I see quite often. The number of people in the sample doesn't determine whether you can form a 99 percent confidence interval (or a confidence interval of any size). What it determines is the size of the confidence interval.

For instance, with a certain sample size your 99 percent confidence interval of a 40 percent estimate might go from 37 percent to 43 percent. But with a much larger sample it might go from 39 to 41 percent.

Best wishes,
Mason

Hi Everyone:

Earlier tonight on Hannity, which I'm sure most of you watch regularly, he had two representatives of pollsters Insider Advantage and The Trafalgar Group. These two polling companies both accurately predicted the 2016 election and both have Trump doing much better than most polls show.

So, why is this? Well, the Insider Advantage representative said that they have, relative to their questions, a "more blended system" without being completely direct to get more information from the respondent than by asking direct questions. He also stated that this approach produces better information and the respondent doesn't feel like there might be retribution if he was to answer directly that he was voting for Trump. It was also indicated that The Trafalgar Group does something similar.

Notice that consistent with what I wrote above, much of the accuracy of a poll has to do with the specific questions that are being asked.

Best wishes,
mason

Great thread.

Yes you can, provided you know what the "correct" results for that variable are supposed to be.

In this election, the number 1 predictor of how people are going to vote is how they voted in 2016. The "correct" numbers for 2016 vote are known from the election results (and from a few other things like demographics data for new voters). If you read the full PDFs for the polls that actual give this data, more than 90% of the people who voted for the two leading parties are going to vote the same way again this time. Any survey that doesn't weight based on this is IMHO trash. Also this is a bit of an insurance against the "shy voters". Someone may be shy about saying they're Trump or Biden voters, but they are also pretty likely to be shy about admitting their 2016 vote too, so they just dilute the "don't knows" rather than removing votes from the camp they're going to vote in).

I've been posting a bit in the sports betting thread about polls. Here are two of my posts I'm vain enough to think are worth a cross-post:

Answering the idea that the 2016 polls favoured Dems:

... so in other words it correctly counted the left and right wings of society, but the more fragmented right (at time of polling) was able to get behind Trump on polling day to get the result they wanted.

On why polling "margin of error" calculated on a statistical basis is actually less of a factor than many think, with a warning about other types of polling error:

(btw the above isn't exactly how you calculate statistical margin of error when you don't know the underlying distribution but it shows the relevant principle)

Malmuth, did Hannity or Hannity's guests cover voter suppression at all when discussing the differences between polling and vote results? Do you have any expertise in this particular issue with respect to political polling?

Mason, what are your thoughts on the below?

There was a decent segment on polling on npr early this morning if anyone's interested.

If you are sure enough in the "correct" distribution of the causative variable to adjust the actual results, then why conduct the poll in the first place? Sounds like you are essentially discarding the results of the poll.

That's the opposite of the point I was trying to make, although I concede I probably gave a particularly poor example/analogy to demonstrate it. The point of that example was that if the results disagree with my prior assumptions, I should either discard my prior assumptions, or I shouldn't have bothered conducting the survey in the first place, since adjusting for prior assumptions in this case it equivalent to essentially discarding the results of the survey.

No. These groups are obviously bad. Trafalgar released a poll yesterday that showed Michigan was close (maybe that happens) but had Trump winning 44% of the Michingan black vote, which is obviously wrong. While it's good to be occasionally surprised by polls and especially crosstabs given the small sample size, these particular polls simply tell Republicans what they want to hear. Rasmussen had republicans winning the house popular vote in 2018. Dems actually won it by the largest margin in history.

Especially when dems lead the generic ballot by.... +7 .

Did you read the next paragraph?

You don't look at the final results till you have your representative sample. Your sample should be representative in terms of the things that matter. You also have to know exactly the numbers what you're trying to be representative of, using known data e.g. from censuses and past election results.

I probably agree with you on the problems with "party identification" being a bit too fluid, ill-remembered, subject to conscious or subconscious suppression if the voter is no longer behind that party (since they last voted or since they registered to vote in a party's primaries). So I wouldn't weight with "party identification" but i would weight with "previous vote" because previous vote is known.

It's probably not possible (within reason). There are broadly 2 types of error when it comes to polling, random and systemic. Random error is super well behaved; a known quantity and can be reduced simply by increasing sample size according to math easily understood by high school students. Systemic error is, on the other hand, virtually impossible to treat from a pure math standpoint (AFAIK) and doesn't go away in a very predictable nature by simply increasing sample size. The possibilty of systemic error likely prevents any practical poll or model from saying Biden (or Trump) is 99% to win even with a relatively huge sample size.

Edit: I just read Lektor's post where he quoted himself. He explained the above concept better than I did and with more detail.

Forecasting the US elections
The Economist is analysing polling, economic and demographic data to predict America’s elections in 2020