Open Side Menu Go to the Top
Register
Expected value of stat with small sample size Expected value of stat with small sample size

12-02-2015 , 04:54 PM
I have a certain stat. It's population average is 60%.
I have player with sample 2/5, so 40%.
How much do I expect this player to actually have in longrun? It's obviously gonna be somewhere between 40% and 60%. The larger the sample, lesser the expected value. With sample size of 1000, it is pretty much 40%, but with small sample it's gonna be closer to 60%.
How do I calculate this?
Expected value of stat with small sample size Quote
12-02-2015 , 05:23 PM
For a binomial measure such as showdown percent, to be C% confident that the sample mean is within d% percentage points of the true value when the true value is believed to be P, the required sample size using a normal approximation is

N = (Zc^2)*P*(1-P) / d^2,

where Zc is the standardized normal deviate corresponding to a C% confidence interval. For 80%, Z = 1.28; 90%, Z=1.645; 95%, Z = 1.96.

For this problem, P= 0.60. Suppose you want to be 95% confident that the sample percent is within 5% of the true value. Then

N = (1.96^2)*0.60*0.40 / 0.05^2 = 369
Expected value of stat with small sample size Quote
12-02-2015 , 05:46 PM
Ok so you told me, that I need sample size of 369 hands, to be pretty well sure about my stat being precise. Right?
But I will never have that sample size. My sample is 5 hands and I want to know how relevant that is compared to the population average.

Lets make a guess. I see someone who has 2/5 (and the population average is 60%) . What do I think his longrun value would actually be? It's gonna be more than 40% and less than 60% for sure. Maybe 55% ? For sample 4/10 it's gonna be lower, lets guess 52%. For 40/100 it might be 43% and for 400/1000 and might be 40,4%.
Those are all just guesses, but you get the point.

And obviously I expect some sort of normal distribution. With standard deviation of maybe 15? That would mean 68,2% players are between 45-75%. 95,4% are between 30-90%. That sounds reasonable for most stats right? Doesnt matter if cbet flop or percentage of win at showdown after bet being called.
Expected value of stat with small sample size Quote
12-02-2015 , 06:29 PM
This is a situation where Bayesian reasoning is directly applicable. As you probably know, Bayesian updating requires some measure of your "prior" beliefs (in this case the 60%, but also how strong your beliefs are).

Using a beta prior distribution, the posterior distribution also is beta. The posterior mean is the ratio of the updated fraction of successes where the strength of your prior belief is reflected in how many "prior samples" you have.

For example, if you are very sure of your prior belief that your opponent's percentage is 60% (or whatever), then observing one more data point will not change your view much. Of course, on the other hand if your prior belief is pretty weak, then one (or a few) observations can nudge your updated views quite a bit.

I recommend fiddling around with the hypothetical prior sample size N that best reflects the strength of your belief. For example, how many hands you have previously played versus this opponent, etc.

Then your best updated percentage is simply ((P)(N)+S)/(N+T) where P is the prior expected percentage (in your case 60%), N is the prior sample size, S is the number of new "successful" observations you observe (e.g. 2), and T is the number of new observations you observe (e.g. 5).

In words, you simply update the percentage where you treat your prior as having N samples.

Hope this makes sense.
Expected value of stat with small sample size Quote
12-02-2015 , 07:27 PM
I like that formula
((P)(N)+S)/(N+T)
That looks like something I expected, except one thing. I don't know N.
There is nothing like my prior sample size. I just have this 2/5 sample on this opponent and I know that population average is 60%. And I can do an educated guess on the standard deviation. Obviously if the standard deviation was very small, like 1, that would mean 68,2% players are between 59-61% , so any stats I have on certain player are almost worthless. Small SD equals high N and high SD equals low N.
I can guess the SD, because I know how normal distribution looks like and I have some sort of idea of how often I see someone with some stat 40% or 80% when 60% is the average, I have no idea how to guess the N. So I need to transfer my SD to your N. How do I do that?
Expected value of stat with small sample size Quote
12-02-2015 , 08:46 PM
The "N" method is basically a heuristic that is often used in practice. Alternatively, you can use analytics.

Analytically a beta distribution has two parameters (call them A and B). The mean of a beta is A/(A+B) and the variance is (AB)/(((A+B)^2)*(A+B+1)).

Using this as your Bayesian prior, if you know (or are willing to estimate) your mean and variance (square of standard deviation), you can solve for A and B using the above formulas.

Then the Bayesian posterior distribution is given by its two parameters A' = A + K and B' = B + T - K where there are K "successes" out of T new observations. Mean and variance of the posterior distribution are easily derived using the formulas above.
Expected value of stat with small sample size Quote
12-02-2015 , 09:33 PM
My resulted N for mean 60% and SD 15 is 3,75, which looks like it could be the correct result.
Using excel I calculated few expected values for different instances/opportunity ratios:

0/0 = 60% (obviously )
1/1 = 68,4%
2/2 = 73,9%
3/3 = 77,7%
5/10 = 52,7%
20/25 = 77,4%
40/40 = 96,6%

Looks realistic imo. Thank you very much for your help.
Expected value of stat with small sample size Quote
12-03-2015 , 02:46 AM
My pleasure to contribute to this type of thread and respond to this type of friendly OP.
Expected value of stat with small sample size Quote

      
m