Setting aside manually evaluating all 1225 pairings of states, my two initial thoughts are to weight on some measure of distance (either by counting how many state borders are between any given pairing or by actual distance) and on state PVI. Both of these have obvious shortcomings. The former discounts the fact that the northeast and west coast are a lot more similar to each other than they are to a lot of states in closer proximity. The latter will obviously create some strange pairings of states it thinks should be similar. While not perfect, combing the two of them somehow should give you a workable, but very rough, first attempt.

Both of those seem like decent methodologies, but I guess I have no way to estimate how PVI or distance should figure in.

I feel like the correlation model should eventually end up looking like (national correl) + (regional correl) + (some kind of demographic similarity?) but I haven't thought of a good way to estimate data driven values for those 3 variables.

On the plus side I'm only considering swing states, sure if Obama loses CA he's definitely losing NH but if Obama loses CA he's getting his ass kicked and the election just isn't interesting to model. So I'm "only" looking at 12 states worth of combos but still.

10 polls all showing the exact same margin for Romney would be odd whatever that margin was. But it appears that you think a 3 point average lead over 10 polls produces a near 100% chance of winning. That doesn't seem to jive with recent history. I know this is a Canadian provincial election, but in Alberta this year the incumbent government trailed in every single poll (22 in total) for nearly a month before the election, mostly by high single or low double digit margins. The last poll had them only losing by two, but no other polls indicated a tightening. However, they ended up winning by 10 points. I'm guessing your model would have considered this result to be something like a one-in-a-billion event.

You're missing #3 - the polls are systematically biased.

Just going off intuition, but not a whole lot of theory, I'd say that your correlation numbers should all be between like .05 and .2 or something, so I'd just create and index using the two (scaling them both so they are the same order of magnitude) and then normalize values to fall on the interval of .05-.2

Depending on how long you plan to be working on this, about a month from now, I'll have access to some insanely good data and can probably get you a whole host of numbers you'd be interested in for fine tuning this model

Canada though. Multiple parties blah blah. Not saying you don't have a point with..

Do you mean
a) polls have house effects (rasumussen is consistently biased towards republicans)
or
b) polling is biased

For the former I agree and hopefully keeping a large aggregation will mitigate that some.

For the latter I think you're better off saying Polling may be biased (undersampling of cell phones, oversampling of people willing to stay on the phone and talk to a pollster, etc) which could be true but I don't really know how to quantify.

I may end up adding just extra noise/error to account for that kind of thinig but I don't have a feel for the scale (if it's there at all), and I don't know how to calculate it using data.

But I don't even really have a way to verify what kind of correlations exist and how big they are at the moment. Like for example teh distance thing would not work with OR and ID. Plus no reason to think TX and OR are more alike than OR and FL. I think for that approach I'm better off going with some kind of defined region but again, we keep coming back to scale.

the part of the reason to include PVI is because of oregon is a lot more like Washington than it is like idaho.

as far as scale goes, i stand by my thought about .05-.2. Every should be positively correlated. Its really hard to imagine that as different as they are, a single event would move oregon one direction and idaho the other. The more likely case is that they are either weakly correlated or at worst, not at all correlated. I would argue that no states are totally uncorrelated because there are plenty of demographic groups that are the same across even extremely different states.

What are you defining as "middling" and "very high?"

Great thread here.

cool stuff, will follow this thread.

I plan to update it every week until the election. Odds are I won't get tired of it but who knows.

can you explain the reasoning behind using teh sqrt

There's always going to be some polling bias. No-one can never get a completely random sample. The main risk is that a large number of pollsters are systematically biased to a significant degree.

Also, there's the whole difficulty of working out the likelihood of a voter turning out even if your sample isn't biased, as well as the possibility of voters being systematically misleading in their intentions (like how the Bradley effect, even though it didn't eventually materialise, should have made Obama's 2008 win less certain than polls indicated).

It's difficult to quantify, there isn't really a way around that. You could perhaps gain some sense of objectiveness by evaluating how polls have historically performed, but there are some things you're just going to have to basically guess.

I don't think that the white religious conservatives in southern Ohio favor Romney enough to really care.

But, on second thought, they DO hate them some B.O.

I definitely agree every state is correlated with every other state to some degree. (Those who disagree see: 9/11).

In this week's update I explored a high/mid/low just for funsies.

Simply every state correlated with every other state at .5 and .9 respectively. This week I added in a .2 scenario.