Quote:
Originally Posted by Matt R.
Had a discussion with a colleague today. Is the following valid?:
We have a distribution of data, that does or does not come from a given probability distribution. Perform a chi^2 test comparing the expected frequencies and the observed frequencies. If the p-value is above a certain threshold (what threshold and how do you choose it in this case? This is one question I had.), then we can say the observed values come from the probability distribution.
It's kind of the converse of how a p-value is typically used. If p <0.05 (or whatever) we can conclude it is likely the data does NOT come from the distribution. But if p > something, we can conclude it does (? -- this is where I don't necessarily agree).
I don't think this is valid reasoning but I'm having trouble explaining why as my background is not in statistics. In particular, I'm curious that if we CAN conclude it comes from the tested distribution, would say, p > 0.95 mean there is a 95% chance it does come from the distribution? What would be the proper p threshold for this (certainly not p> 0.05, right?)
I've never seen a p-value interpreted this way and it doesn't seem right, but wanted some feedback from someone more formally trained in statistics. Thanks!
p-values are used to reject a hypothesis, not accept it. When you get p < 0.05, what you're saying is that the observed distribution is probabilistically rare given the assumed distribution. Given a distribution, what are the chances of the data?
You can't turn it around because the structure of the question is backwards. Given the data, what's the probability of the distribution? To answer that question, you need a way of measuring the space of possible distributions, which p-values can't do.
For some more intuition on that, it's worth noting that even if you knew with absolute certainty what the underlying distribution is, you would only *expect* to see p > 0.95 about 5% of the time because of how it's calculated. So using this metric, you would only have a 1 in 20 chance of confirming something you were absolutely certain about. And so by that measure, this would be a terrible tool.