You're framing it as some formula that some experts worked out. But what if you personally have worked out the probability of hitting a flush from basic math? What's more likely, someone hitting a 40,000 shot or math as be know it being wrong?

OTOH, if Nate Silver tells you his model, based on prior elections, gives Politician A 40,000:1 odds and he wins, you're very likely to think the model is wrong.

Exactly. Probability is always a measure of what we know about the situation. Asking probability questions about mystery black boxes gets you nowhere and isn't really what happens in the real world.

If the competing theories are even money versus not even money, you need two trials to learn anything.

Not true.

Say you have H1 that says that the a binomial 0/1 process is even money and H2 just says it isn't (notice that saying "isn't even money" is not enough to properly define H2, but we will ignore that for now).

At the first trial we get 1. While it's true that P(H1) and P(H2) stay unchanged, the "shape" of H2 changed: we know that if H2 is true and the event is not even money, it's more likely that it will tend towards 1 rather than 0. This is an important result. It would also be totally irrational for instance to bet on 0 at even odds if we had to bet the next trial. Do you really think we didn't learn anything?

If we really hadn't learnt anything after trial 1, we wouldn't learn anything even after trial 2. DUCY?

A model can not exist without some sort of confidence level in its prediction.

You say something is 1000 to 1 happening. Where did you get the 1000 to 1??? The 1000 to 1 has to come from somewhere. All the models are based on assumptions. Your confidence in the assumptions gives you the answer you seek. If you dont know the assumptions, it was nonsensical to predict 1000 to 1 in the first place.

If something is 10000000000 to 1 according to you, and it happens on the first trial, common sense tell you your model is wrong. But this "common sense" is, like David says, is based on a empirical distribution. During your lifetime and the lifetime of your ancestors your brain has seen /made many predictions, and it stored how many were right/wrong. This is what creates the common sense. There is no easy math you can substitute for this. If you write down all your 1000 to 1 prediction in the future you can see how many times you were right. Then next time you can use this data to do the math.

In the Hillary Clinton 1000 to 1 bet for me personally the model is 99% to be wrong, because it was based on the bull**** assumptions that 1.no people are gonna change their mind in final days leading to the election2. people speak the truth in surveys.
Furthermore there were many different models out their that predicted a higher chance for trump+some good stats blogs and people like Nassim Taleb were debunking the 80%+ clinton predictions prior to the election.

But if I go to roulette table at my casino and say "Lets say if it spins 17 two times in a row, odds are 1 in 1296" and then it comes up 17 two times. Im still like 98% sure the odds were 1 in 1296

The scenario were you have no info other than "its 1000 to 1" can not exists, because you wouldnt have been able to make the prediction with no further info

Another observation: The human intuition evolved to predict probabilities for events like:

- Is this blueberry poisonous?, Is this girl a good mother for my children?, will that animal be able to kill me?

It was not evolved to estimate probabilities for questions like:

- Is this stock a good investment? Are my political views correct?

First type of questions have way lower uncertainty/noise then second type of questions. So, you evolved in low uncertainty world. But now we are in high uncertainty world. Therefore the answer to your question: "What is the chance my model is wrong" is probably a lot higher then your intuition suggests. And you can make better decisions if you try to train your mind to be more flexible about ur opinions and quickly change them in light of new data.

Models exist without confidence levels all the time. Take a physics model of the frequency of existence of a certain particle. It is predicted by your particle theory to appear in 1 in 50,000 high energy events. Or take a deep learned market odds calculating algorithm. It simple spits out a number; you have no idea of how wrong or right it is or how wide the error bars. Or a new sports betting formula put together from your knowledge of how games work.

Even the claims of people are often black box. Take the Princeton clowns above. They claimed 99% and we have no other information except for 99% and a single data point going the other way.

This stuff happens all the time. Given that, what can we say? Is it reasonable to say that the Princeton model is >50% to be substantially overestimating the odds? Is that a robust statement to make given just those two bits of information?

If one theory says the coin is fair and the second theory says that the coin is unfair but makes no assertion about how the coin is unfair (thus implying that the unfairness is just as likely to be towards heads as tails and to a degree [80-20,70-30,60-40 or whatever] just as likely to be helping tails as heads) then the first flip tells you nothing about whether the even money theory is true but the second flip will steer you towards or away from the even money theory. (It becomes more likely if the two flips are different and less likely if they are the same.)

You have some other information like:

-Your experience in general with predictions from humans
-You know the prediction comes from Princeton University. You have some beliefs/experience about reliability of university predictions
-You know its a prediction about an election. You have some knowledge about elections, maybe also about election predictions (Elections are rare event so uncertainty about best way to predict)
-Maybe you read something about their model the prediction was based on
-You have information about other peoples predictions for the same event

with this info its for me its for sure reasonable to say they are >50% to severely overestimate the odds. But thats entirely subjective, someone with different prior beliefs may arrive at a different conclusion.

For all your examples there is some extra information that you will have that can help to tell you how much beliefs should shift. Even if you know nothing about the specific situation you have some general beliefs about how often predictions are right. You know when a 1 in trillion event happens the first trial model is almost certainly wrong because you have seen many wrong predictions in your lifeand the odds of this happening were much higher in case the model was wrong. But if you want to know exactly how likely it is that is the model is wrong you cant say that. You have some intuitive distribution in your head but not the exact numbers.

If it moves the needle in any way shape or form, then you learned something.

In this particular case, if your first trial lands on heads you have learned that the coin doesn't always land on tails. That might not be much, but it is something.

That's what I said. P(H1) and P(H2) stay unchanged, but that's far from saying that we learned nothing. We learned that if the coin in unfair, it's more likely to favour head (if head was the result of the first flip).

Learning nothing means that we could disregard and forget the result. That's not true.

You also learn that the coin isn't too heavy to flip. My point is that you need two flips to change your opinion of the even money theory which I'm pretty sure was what Toothsayer meant.

There has to be a distribution without priors. A heat map of likelihood given a single isolated claim and one contrary event.



The heat maps shrinks with more data points and other information (such as background information on models in general, the kind of stuff you list), but it still exists for one data point.

Take a claimed 1000 to 1. First event is a positive. We have a heat map here of the odds the model is true. Probably strongest around 95% chance the model is a poorer predictor than a 50/50 model.

Just wondering what the math/simulation/graph of this looks like, and if anyone has studied this and written about it. And how to best display and think about it meaningfully.

I mean, given a contrary data point and a long odds model claim, I think we can all agree the heat map of probabilities that the model is better than a 50/50 model does NOT look like this:



It looks something more like the heat map in the previous post, imo. But how to quantify it and talk about it objectively?

Of course I got what you meant. But it has to be stressed that yours was a just a particular example. If the two theories were "even money" vs "head is favourite" (you know for some reason that tail cannot), you would have learned since trial 1. The sentence "you need two flips to change your opinion of the even money theory" in general is plain wrong.

You can have some scenarios in which you have "100-1" against some other theory and build the alternative theory in such a way your posterior beliefs would remain unchanged after flip 1. There is nothing special on 1-1 theories.

Not sure if you have a purpose for this discussion beside "I want to bash Princeton morons; please help me in finding a mathematical formula for it", but Bayes' theorem is pretty clear (sure you know, but I report it here just for clarity):

P(H|E) = P(E|H) P_0(H) / P(E)

There is no other way of calculating P(H|E). You need the prior and you need also P(E) which can be calculated only by considering every possible theory that have a different from zero prior probability.

Sure, a frequentist would just ignore every possible alternative theory and run some hypothesis testing. Sadly, these tests cannot be interpreted as probability and their usefulness is very limited (if any). There is a huge literature on the subject (sure you are at least partly aware of it).

You can also define your own concepts and your own methodologies to deal with your issues. No matter what, the results you will bring won't tell you the likelihood of a model to be wrong, because there is just one way to calculate it and it's Bayes' theorem.

Algebra was pretty clear too until we found integration and limits.

Unfortunately I'm largely ignorant of the probability literature, hence my SOS in this thread.

I simply don't accept that a first-trial hit on a long odds model without further information tells me absolutely nothing about the probability of the model being better than a 50/50 model; that view is wrong regardless of what the literature claims.

Thanks for your patience so far in explaining.

"In general" its right. If you mean by that expression the majority of times someone is debating whether something is 50-50 or not. It is only wrong when competing theories are asymetrical.

Probability theory and statistics are very tricky and require a lot of thoughts. Especially statistical inference.

First thing you need to consider is that there are no shortcuts. You need to formulate precisely assumptions, theories and models. You have to think about any possible explanation. You need to assign coherent prior beliefs for various options. That's not easy.

You are absolutely right when you think that a single data point tells you something. Unfortunately, understanding what it tells can be so complex and it depends a lot on the specific problem you are dealing with.

Let's try to examine model elections. Predictions will depend on the information considered and how that information is modelized. Let's build some models.

1) model 1: I just know that there is an election and one candidate will win. I also know that there are N candidates. If that is my information, my model will conclude that any candidate is 1/N to be elected.

2) model 2: I know what model 1 knows and also that the last M elections were won by candidates of just two parties. My model will conclude that with probability (M+1)/(M+2) (see why?) will win one of the two candidates of the major parties ((M+1)/(2*(M+2)) each). The remaining 1/(M+2) will be divided between each other candidate.

3) model 3: I have also knowledge of some polls. I don't have record of how past polls were able to predict the right result. However, I have knowledge of how this polls are conducted and am proficient in statistic. So, I can, with my expertise, build a prior about P(have a polls result | real risult=x). So, I can process the polls and my model will spit out some probabilities for each candidate.

4) model 4: now I have knowledge also of past polls. I can model more precisely the relation between polls and actual result.

5) model 5: I start to incorporate local polls and the features of the electoral system. I also try to model correlations between results in different regions.

You can go on. Aside from the very simple models, there are lot of assumptions to be made. Some models might be off because relations between variables are treated unrealistically. Other because they miss important and available information. Other because the statistical procedures are not sound.

If you really want to bash Princeton's guys learn how they built their model. Question their assumptions. Find data that disprove their choices. Prove that they treated their data poorly. Show that they choose to ignore something important. Compare the treatment of data with other models and show that there is a better way. In a word, build your own model and prove to be better (or at least contribute to build a better model).

You don't want shortcuts. It's futile to disprove them on general assumptions that, unfortunately, don't hold. A lot of examples were made ITT to explain why. Think about them lengthy. The simple fact that a 1% event happened, by itself, doesn't assure that likely they are doing a bad job. It might be a reason to investigate, but not to conclude.

Sorry for the long post (might be easily my longest in 2+2). Hope to have given you something to think about. Otherwise, sorry for the time wasted!

Maybe we are trying too hard. However, one last chance for me to express what I meant (sorry again for being too lengthy).
This was the TS'sentence that triggered the discussion:

I wanted to stress some simple points.

1) A single data point can tell something about a model saying 1-1 odds.
2) The fact that sometimes you need two data points is not because one model gives even odds.
3) Everything depends on the competing models. 1-1 models have no special properties in this regard.

So I didn't much get why you raised the point that sometimes you need two flips. The TS's sentence seemed (at least to me) to imply that you need ALWAYS two data points when dealing with 1-1 models. This wrong belief needed to be addressed.

I also want to stress that thinking in terms of "a model" vs "everything but the model" is almost always lazy and very often wrong. So, if I have a model saying 1-100, it's almost always very stupid to compare it with the "not 1-100" model (and that still holds, possibly to just a lesser extent, to even money models).

He meant nothing of the sort.

You've used an implicit prior here without realising it; to come up with the 95% value!

Your prior in this case, is whatever value you'd need to use to give a likelihood ratio of 19...

BUT: there is nothing to say that the particular prior / likelihood ratio you chose is any better than any other - it's just a value that seems to "fit" with your previous observations (see: "No free lunch theorem").

Juk

The USMNT just bozoed qualifying for the world cup as about a 69:1 favorite to survive this night of games. What do we say?

I think you'd be less inclined to rely on the Princeton model for placing your bets in the next election.


PairTheBoard

Seems unfair that we had to play both Trinidad and Tobago at the same time.

Probably better to call the first theory the "fair coin" theory rather than the "even money" theory since it would be an even money bet on the first flip but wouldn't be on the second flip.


PairTheBoard