Quote:
Originally Posted by Gster
Can someone offer a definition of these two concepts in statistics?
I tried looking it up on wiki and I still can't grasp the definitions given
I'm under the impression that the "P-Value" has some kind of relationship with
"alpha"??
As for the "level of significance" when it is said that we require a 5% level of significance, does it mean a 95% confidence interval (so an "alpha" of 5%)?
If my understanding of anything is faulty please correct me as well...
One problem is that the behavioral sciences have completed butchered the meaning and understanding of these terms. I'm sure that is one reason you are confused...totally understandable.
A p-value is a probability or proportion. Strictly speaking, a p-value is the probability that the sample data would occur if a pre-defined "null hypothesis" were in fact true in the population.
That is, a p-value is the probability of getting the data you got, given the hypothesis (null, which is often "zero" by default). In a bayesian sense the p-value is p(D|H).
Many people make the mistake of thinking that the p-value is the probability of the null hypothesis being true. IT IS NOT! People make this mistake and then assume that a p-value of .05 means that they are 95% sure that their alternative hypothesis is true. They could not be more wrong.
A "statistically significant" p-value simply means that the data you gathered (group differences, correlation, etc.) was unlikely to have occured by chance alone (i.e. unlikely to have occured if the null hypothesis were in fact true).
Let me emphasize: A STATISTICALLY SIGNIFICANT P-VALUE DOES NOT MEAN YOUR RESULT IS "REAL" OR EVEN "IMPORTANT".
But a small p-value does mean that your result (observed data) was unlikely to have occured if in fact the null hypothesis were true at the population level (e.g. no relationship between the variables in the population, no mean differences in the groups at the population).
There are some major problems with reliance on p-values that intelligent behavioral scientists have long identified. The first of which is the fact the "null hypothesis" is often assumed to actually be a "nill hypothesis". That is, null often = no effect whatsoever. In the behavioral sciences however, this is an absurd notion. Almost everything is related to everything. Almost every group mean is different to some decimal place. For the "null/nill" hypothesis to be true, one would have to prove that two groups are equal to an infinitely long number of decimal places. I don't have the time to walk that far.
So it is quite odd that behavioral scientists (like me) would be even remotely interested in knowing the probability of getting their data if in fact the null/nill hypothesis were true, when in fact we know it almost never is.
A second problem with over-reliance on p-values is that people tend to believe that effects with p-values below .05 are "real" and anything above ".05" isn't. The .05 cut-off is completely arbitrary. And ultimately decisions about whether an effect is "real" or "important" come down to the researcher and his/her research interests.
In any case, the .05 level is often chosen because in a signal detection theory sense, it means that there is a 5% chance of making a Type I error. A type I error means that you reject the null hypothesis (say there is an effect when in fact there is none). Now as I have mentioned before, the notion of Type I error is pretty much absurd in the behavioral sciences. But stay with me anyway. The probability of making a type I error is called an "Alpha Level". Likewise, there is a probability called "Beta" which is the probability of making a Type II error (saying there is no effect when in fact there is an effect).
As an example, consider this scenario. Person A either has HIV or doesn't. A test for HIV is either positive or negative. If Person A doesn't have HIV and the test is negative, we are correct. If Person A has HIV and the test is positive, we are again correct. However, if person A doesn't have HIV and the test is positive we are incorrect and have made a Type I error (said there was an effect when there wasn't one). If person A has HIV and the test is negative we are incorrect and have made a Type II error (said there was no effect when there was one).
In any case, alpha is typically set at .05, accepting the fact that 5% of the time one might reject the null hypothesis when in fact it is true.
Anyhow, I'm sure this answer is much longer than you expected. I hope you found it valuable.
Most importantly however, remember that a p-value is the probability that you would have gotten your data if the null hypothesis were in fact true in the population. It is not the probability that the null hypothesis is true.
Sherman