This should be an easy one but it's eluding me.

Take a simple situation. You estimate the odds of an event happening on any particular trial to be a billion to one. It happens on the first trial. What are the odds your model is wrong?

Or a more practical example. On November 5th last year, the people at the Princeton Election Cucksortium estimated the odds of Clinton winning at 99%

Trump won, by almost the reverse. What are the odds that the model is broken rather than just unlucky?

This is easy enough with estimated error bars, but when there are none, how do you arrive at a figure of the odds of something being wrong with any kind of authority?

What do you mean with "the model is wrong"? A model is often made of assumptions and hypotheses. Some of them might be off and some spot on. You need to be more specific. You likely want to know "how good is the model in predicting", but is a different question. Know that every model is wrong in a way or another.

Don't think you are looking at it in the right way. Models are just descriptions of reality and are not reality itself. They are all broken. It's more useful instead knowing how to measure the goodness of a prediction. There are several ways to do it. The most simple is to calculate the likelihood of what actually happens over several predictions and compare the result with competing models.

Some caveats. The competing models must be transparent: they must indicate their inputs, how they work and be clear on the results. If you have several models, you can amass predictions and then the likelihood of actual events according to each model. The highest the likelihood the better the model.

Regarding the election, what we can say is that Nate Silver's model are likely better than the one you quoted (they gave Trump an higher chance). That's all we can say for now. But, as every model, both models are wrong.

Agree with you on models of course.

Alright, let me put it another way.

What can you say about the true probability of an event from a single trial? There has to be something we can say other than "nothing".

How much does a single positive event reject a long odds hypothesis? If I say something is one in a billion, and it happens on the first trial, how much less confidence should I have in my one in a billion hypothesis? Common sense says a great deal less - what does math say?

There's no real sensible way to look at this without thinking about what goes into the modeling and how much we trust that and making Bayesian desicions.

There is no concept of "true probability", especially if we are considering events that happens just once like the result of an election. It just depends on the state of information you got. If you knew exactly each person thinkings, feelings, emotions and how they evolve approaching the date of the election you could predict exactly the outcome.

Regarding your question, you must be more specific. Declare a model (saying that something is 1 in a billion to happen is not a model), describe the information you got and state the actual result (you can totally make up an example). Then, we can start a discussion.

With regards to the election, Hillary had it stone cold locked up until Comey came out like a week before and said she was still under investigation. So maybe the odds of everything unfolding the way it did were actually 1%. We are all living in this ****ing reality tv show world now, all thanks to a bad cooler.

Also, I'll have to find the link, but I read an article from some university statistics professor saying that Hillary winning the popular vote by 3 million and losing the electoral by essentially 70k votes was a major statistical anomaly. Considering how it is now known that Russian trolls paid for ads on Facebook and targeted them at extremely specific demographics of people in Michigan, Ohio, and Pennsylvania (aka the states that swung it to Trump), I'd say the models left out at least one major factor: Russian meddling.

A scary amount of people actually believed (probably shouldn't put that in past tense) that Hillary and other dems were running a child sex ring under a pizza shop in Pennsylvania. So ya maybe the models can't model for how dumb people really are.

Chance a planet has intelligent life; 10^-23 or less (my model). Chance earth has it; 100%. Come on now. Chance the visible universe develops intelligent life if we vanished ; over 99%.

The model is astronomically perfect actually. What is the chance the popular vote is +3mil and you lose because of how some 70k key morons voted in 3-4 key different places. Of course less than 1%.

Their model was perfect and still is. Lets run it 100 more times since October 1st and see what happens. Oh wait we cant. And that will prove the tragedy of the century. Of course Hillary also owns partially that loss for who she is and how she played all her life. But a big f u to those particular Russians that made it possible and the American traitor mfers that helped them do it.

Your first trial is not an independent event, it's selection biased.

I don't want to rehash politics. It was just a very prominent example of predicted long-odds going the other way.

I'm just wondering what we can say about the chance of long-odds models being correct, when the supposed long-odds event happens on the first trial. Clearly we can say more than nothing - if a model is 100 trillion to one and the first trial hits that, you can say that the model is over 50% to be wrong. Can you say it's over 99% to be wrong? 99.999%? There has to be some math on this.

We can certainly say something here. I'm just wondering what. I don't accept that a data point that is modeled to be 1/1,000,000,000,000 to hit, hitting, means we can say nothing about the veracity of the model because we have only one data point.

Again, you must define better your context. Things may change dramatically.

Consider this scenario. There are two hypotheses, H1 and H2. We know one of them must be true. H1 predicts event E to be 1:10^9, while H2 predicts E to be 1:10^13. Event E actually happens. Just use Bayes' rule and see that our belief on H1 actually increases a lot thanks to E happening, even if it was so unlikely.*

In that scenario a single data point is hugely informative. On other scenarios it might be insignificant.

Some considerations.

- Consider always alternative hypotheses. An unlikely event by itself doesn't imply the hypothesis being unlikely unless other hypotheses assign the event an higher likelihood.

- A model is built on a set of hypotheses. If predictions are off, you need to test single assumptions and see if you can improve the model.

*That scenario, even if much less extreme, is what indeed happened for election models. All the most serious models gave Clinton favourite. However, Nate Silver's model gave Trump the higher chances and Trump winning has cemented Silver's model as likely the best (even if people not understanding probs might consider the model a failure).

The point i was making is that a model that is more fine tuned to reality (has access to more data) can predict earth is not 10^-23 (like a random rocky system) but way bigger because of all kinds of good things that happened to this planet (like size, proximity to sun, satellite, type of star, chemical composition , distance to gas giants etc) upon very close examination of local conditions and properties of the system. So eg if one had access to what Russians were doing and how close the results were in some states and what this mfer Comey were about to do without any credible results at hand that mattered in anything (yes he is a loser either way, you do not play such bs games near election and then claim you are ethical, you are a loser that wants attention) then the model would update differently than 99%-1%.



You need to do some Bayesian updating of the model in your general question after the fact. If the model has some natural probability to be wrong in all its construction, this probability is enhanced now in first trial.


Like say you have 5% chance the model is 50-50 crap claims and 95% it predicts well the result with 99%-1% chance. Now the chance the model is trustworthy starts at 95%. After it has had one trial that is bad it will go to what?

P(Model accurate|1 trial gave the long shot)= P(1 trial gave the long shot|Model Accurate)*P(Model Accurate)/P(1 trial gave the long shot)


P(1 trial gave the long shot|Model accurate)=0.01

P(Model accurate)=0.95 (prior assumption of model's respectability given whatever historical records or what not)

P(1 trial "long shot"| Model not accurate)=0.5 say (if model is bs then say its 50-50 in predictability although could be other also)


P(1 Trial gave long shot)= P(model accurate)*P(1 trial long shot|Model accurate)+P(model not accurate)*P(1 trial "long shot"| Model not accurate)=

= 0.95*0.01+0.05*50%=0.0345

So

P(Model accurate|1 trial gave the long shot)= P(1 trial gave the long shot|Model Accurate)*P(Model Accurate)/P(1 trial gave the long shot) =0.01*0.95/0.0345=27.5%

So chance model is correct given the 1 trial drops dramatically to 27.5% only.


You can of course go to other alternatives (instead of the good or 50-50 crap options) like not correct to claim 99% but more appropriate to claim 80% (ie model not accurate but still reasonable direction- i mean in real life you have many alternatives rather than forced to choose between just 2 extremes 95% accurate and 5% coin toss crap ) etc.

See also
https://en.m.wikipedia.org/wiki/Bayesian_inference

or

https://en.wikipedia.org/wiki/Bayesian_inference

Now try even more complex updating like probability model is right given what happened as in loss but also events before election and results close in key states. if these start looking like 1-2% rare surprises and manipulations then the model gets a much better image moreover the loss because that 1% is precisely happening that way.

No offense, but your model is rubbish. No one has any clue what the odds of that are.

No what is rubbish is your understanding of my model. It nicely explains why the universe looks that way and not full of aliens already having colonized us and why the age of the universe is not way bigger than our solar system's age. We are simply first within a few billion light years . It is a very simple sensible argument. The next one is simply very far away.

I don’t know if masque’s alien model is correct or not, but his political model surely is not. Seeking to isolate a single cause in a complex election in which literally millions of facts play a role over years leading up to a result is a fool’s errand.

Well of course its chaos out there (and i have argued that too if you cared to read and not all you guys come here to attack others) but given the usual chaos you can do little about (that you have in each election) we definitely can do something about some mfing improper action like that of Comey so close to the event or the hacking of a superpower's future by other powers and the manipulation of the voters by trollers and bs warrior leakers. We can know for sure that these 2 are enough to move at least 70k morons. And you wont have a chance in hell arguing against this fact without violating the very logic with which elections campaigns are designed.

First, masque, I’m not attacking you. I am sure I am on record as defending criticisms of your posting in the past. Vis-a-vis Comey in particular, it is wholly clear that he soft-pedaled the HRC investigation and sought to handle it in a way that would not damage HRC. So I agree that he is an ahole that politically influenced the election, but he did it in both directions. Take whatever view of the politics you want to make yourself feel better, but you should realize that the analysisnhere is not up to your usual rigorous standards.

It’s like blaming the outcome of a basketball game on a single missed free throw at the end. The outcome is instead determined by the entirety of the 150 or so shots taken.

In any event, this is the end of the derail for me. I thought OPs question was interesting.

My model says the odds of alien life on a planet are 10E-1,375,384,469,038,046. It has as much empirical basis as yours.

Well, that’s just clearly incorrect. It’s estimated that there are 10e24 stars in the universe. The fact that life exists on earth makes it waaaaaaaaay more likely than the thoughtless number you assert.

But this goes back to the OP: you can't know it's wrong simply because a longshot made it.

Or, to use the other example, we don't know the election forecasts were wrong just because Trump won.

My contention is that you can, however, if the odds are far enough out. Look at masque's math using Bayes. Pretty damning to a 99-1 model even with generous truth likelihood priors.

I was aware of this Bayesian analysis, but I'm just wondering if we can create a probability distribution without assuming anything about the priors, as masque had to do. A kind of sum or integral of all reasonable Bayesian priors.

Like if a model says 1-1 odds, and we know nothing else, and there's one data point, we learn zero about the correctness of the model.

If a model says 1000-1, and we have the first data point hitting that 1000-1, we can say a lot about the correctness of the model. In fact, we can conclude it's very very likely incorrect.

Is there a graph that has been drawn up of these probabilities, or code/formulas I can put into graph form?

So if you see someone get dealt two Royal flushes in a row, do you conclude that the formulas people use to predict poker odds are wrong?

If I see ten heads in a row on a flipped coin, I certainly investigate whether it’s loaded.

If you know that the problem is not pure math and that experts make something a billion to one underdog, you can come up with a chance the experts were wrong (if there is a success on the first try) if you have historical data about other billion to one predictions that subsequently were honed in on. In other words if one out of ten thousand billion to one predictions that subsequently could be shown to be either approximately true or way too high, were in fact way too high, then it would be about a hundred thousand to one favorite (given this info and nothing else) that the model was wrong when it hits on the first try.

David: yeah I think the most insightful priors here are the error-proneness of the modeller's mind/procedure. Which is very high; people just stink at modelling.

But as you say, with high enough odds, even a tiny incidence of failure throws out all the models that hit contrary data points.
If you've never seen another hand played, yes.

There's a new game called Trolly Poker. Experts work out formulas, and claim that a Trolly Flush - the highest hand - is 40,000 to 1. The first and second hands are a Trolly Flush. Assuming the cards are shuffled and dealt according to the rules, you conclude to 99% probability that the formulas are wrong.

Not true. (BTW, I've never seen a context in which someone is giving some odds without "knowing nothing else". You pretty much always know something else).

Lol, no.

See this.

Scenario 1: you have a coin and you know that either it is regular or it has two tails. So you have two models: the first says 1-1 odds head-tail and the other says 100% tail. You can just see the result of a flip. It is head. Do you think "we learn zero about the correctness of the model"? (note that we learn something even had the flip been tails).

Scenario 2: I have a fair RNG in the [1-1000] range which I trust 100%. I use it to generate a number and it's 335. It was 1000-1 to be exactly 335. We really "can conclude it's very very likely" that the RNG is not fair?

Think also poker, as someone already pointed out. My model says that any 5 card combo is 2598959:1. According to you, no matter what hand you receive, my model is likely incorrect.