The idea for putting all this in a post came up after 1p0kerboy and I were hastily explaining the same calculations to each other at once in a thread... as unclear as I may have made some of the info. here I'm expressing this more clearly thanks to the discussion with 1p0kerboy.


In this post I’m going to go over a few statistical applications to better understand sample variance. I’ll explain how to use information from PokerTracker to estimate intervals that your “true” winrate is in with any given level of confidence. Before I get started I’m going to include a brief disclaimer/rant that’s really more of a technical point, you should feel free to skip this section if you’re not already somewhat familiar with the idea of measuring variance quantitatively and familiar with the normal distribution… the upshot of this point is that the math that follows is a [probably slight] underestimate of your real variance.

\begin rant
Your profit/loss from a single hand of poker is a random value, but it is certainly not a normally distributed one. The distribution of our winnings from the hand does, however, have some average value. So the Central Limit Theorem assures us (so long as each hand’s outcome is independent and identically distributed) that if we consider samples of the aggregate winnings of a large number of hands, that each of these samples outcomes is in fact a normally distributed random outcome.

While anywhere from 30 to 100 events summed together is typically considered enough for the Central Limit Theorem to apply, poker hands are a bit more extreme, with large events (+/- 100 big blinds) occurring with a frequency that I don’t believe 100 hand samples are very close to normally distributed. The SD of 100 hand samples provided by PokerTracker underestimates our true variance. For example, using the numbers used later in this post, the probability of winning 4 buyins (200 ptBB) in 100 hands is approximately 1 in 5 million.

Finally I point out that since hands are neither entirely identically distributed (the distributions vary with position relative to the button at the very least) nor independent (depending on our current opponents we have many hands clumped together with expectation and variance higher or lower than other sets of opponents). These departures from the assumptions of the Central Limit Theorem lead us to again (slightly) underestimate our overall variance.
\end rant


Background Info:
Before you can apply the information in this post, you need an estimate for your winrate, the number of hands this sample is over, and some measure of the standard deviation or variance of your winrate over this sample. In PokerTracker, you access the first two numbers on the “General Info.” tab. To find your standard deviation in PokerTracker, click on the “Session Notes” tab, then click “More detail…” In the pop-up, the white text contains “Standard Deviation/100 Hands:” which it will list as both a dollar amount and a Big Bet amount. It should be in the neighborhood of 20 to 200 Big Bets, depending on your playing style and type of game you play in.

I’m going to assume you already understand what the winrate means – bear in mind that it is most commonly measured in BB/100 hands where BB may stand for either big blinds or big bets (twice the big blind). The latter is more conventional despite making less sense for NL/PL games, and I will assume big bets in my calculations (for clarity, I will always write ptBB/100 to indicate PokerTracker big bets). To toggle which PokerTracker uses while displaying statistics, go to the Preferences tab and on the right hand side halfway down check or uncheck “treat ‘BB’ For NL/PL As Big Blind Amt.”

The standard deviation (SD) statistic provided by PokerTracker tells you the SD of a 100 hand sample. The SD is a measure of the variation about the average result – the larger your standard deviation, the more likely it is that any random sample of 100 of your hands will deviate largely from your average winrate. We assume that for large numbers of hands your overall winnings will follow a normal distribution (http://en.wikipedia.org/wiki/Normal_distribution). Probabilities based on the normal distribution are well understood and we make use of a Z table (here's one: http://lilt.ilstu.edu/dasacke/eco148/ZTable.htm) to calculate probabilities of various outcomes of normally distributed random variables.

To give you a quick impression of the normal distribution:
Assume your total winnings of each 100 hand sample are normally distributed with, say, an average profit of $15 and a standard deviation of $100. Then when you play 100 hands, approximately 68% of the time your winnings will be within $15 +/- $100; that is, between -$85 and +$115. Similarly, about 95% of the time your winnings will be within $15 +/- 2*($100), or between -$185 and +$215. And about 99.7% of the time your winnings would be within $15 +/- 3*($100). Knowing the average and SD of each of our 100 hand samples, we can use a Z table to find the probability that the actual winnings of any given 100 hand sample will fall within a given interval (this is where the percentages 68, 95, and 99.7 come from… later in this post I will explain how you can do that sort of thing yourself).

The thing is, we don’t actually know what our “true” winrate is. If we play 100 hands and win $15, we may guess that our winrate is $15 per 100 hands, but if we’re going to make a guess like this we’d like to know how likely it is that our true winrate is close to this guess. If we happen to know our true SD, we can build confidence intervals… Assume that our true SD is $100 per 100 hands. Then a 68% confidence interval for our true winrate per 100 hands would be (-$85, +$115), a 95% confidence interval would be (-$185, +$215), and so on (these intervals are so wide because 100 hands is such a pitifully small sample). An important point to stress here is that the level of confidence (68%, 95%, etc) is NOT the probability that our true winrate is actually within this interval. This sort of probability is something that we can’t actually know without making much more complicated assumptions and including more background knowledge. What a 68% confidence interval of (-$85, +$115) does tell us is that if our true winrate were below -$85 per 100 hands (or above +$115 per 100 hands), then in any given sample of 100 hands, having a result of as good as +$15 (or as bad as +$15) would occur less than 32% of the time (100% - 68% = 32s%).

Calculations:
The SD of each of our 100 hand samples is something which converges much faster than our winrate does, and after a relatively small number of hands (still of course in the thousands if we want much reliability) we can assume that the SD found in PokerTracker is a very good approximation of our “true” SD for 100 hands.

Next we’d like to find the SD of our entire hand sample… as an example, let’s say that we’ve played 138,200 hands, and PokerTracker tells us that our SD for a 100 hand sample is 38.8439 ptBB. For those of you not interested in the derivation at all, you can skip to the bolded section below. Since our sample of 138,200 hand sample is the sum of 1,382 samples of 100 hands, it may be intuitive to just take the SD of 38.8439 ptBB and multiply it by 1,382… SD’s don’t quite work in this additive way. But the variance of our samples does have this property. Variance is (by definition) the square of the SD.

So the variance of a 100 hand sample is equal to (38.8439)^2 = 1508.85 (ptBB^2). The variance of our entire 138,200 hand sample is then equal to 1382*1508.85 = 2085230 (ptBB^2). We can then take the square root of this number to find the SD of our 138,200 hand sample: SD = sqrt(2085230) = 1444.03 ptBB.

Since we measure our winrate in ptBB/100, it would be nice if this sample SD we’ve calculated were in the same units. Our SD of 1444.03 ptBB is over 138,200 hands, so changing units, we have SD = 1444.03 / 1382 = 1.045 ptBB/100.

As shortcuts we can remember that the SD (in ptBB) of the 138,200 hand sample is equal to the SD of the 100 hand sample times sqrt(1382), and if we want the SD of the 138,200 hand sample in ptBB/100 instead, we take the SD of the 100 hand sample and divide by sqrt(1382):
SD = 38.8439 * sqrt(1382) = 1444.03 ptBB
SD = 38.8439 / sqrt(1382) = 1.045 ptBB/100

We are now equipped to accurately gauge the error of our winrate estimate. Suppose that in our 138,200 hand sample we won a total of 4422.4 ptBB, for a winrate of 4422.4 / 1382 = 3.2 ptBB/100. We calculated that the SD of this 138,200 hand sample is 1.045 ptBB/100, so a 68% confidence interval for our winrate is (3.2 +/- 1.045) ptBB/100, a 95% confidence interval is (3.2 +/- 2*1.045) ptBB/100, and so on. If you just want to remember that you can form a 95% confidence interval for your true winrate by taking your observed winrate and subtracting/adding 2 SDs, you’ll have a good idea of what your true winrate may be.

The remaining discussion requires that you make use of a Z table (http://lilt.ilstu.edu/dasacke/eco148/ZTable.htm). If we wanted to find a 98% confidence interval, we would look up how many SDs away from the average we would have to go in each direction to include 98% of the normal distribution. Typically when we find a 98% confidence interval, we would want the 2% of the distribution that we don’t include to be evenly split – 1% at the very top and 1% at the very bottom. So we would look up on a Z table how many standard deviations above average we have to be so that only 1% of the distribution remains; i.e. we find the Z value which leads to .99 on the table. This value is 2.33 (and -2.33 is the Z value with only 1% of the normal distribution below it), so continuing from our same example, a 98% confidence interval for our true winrate is (3.2 +/- 2.33*1.045) ptBB/100.

Now let’s look at a similar problem, we want to find out with which level of confidence we could be sure that we’re a winning player. We basically want to know what our confidence level would be on the interval (0, infinity). This amounts to finding out how much of a normal distribution with mean 3.2 and SD 1.045 lies above the value 0. To do this, we find out how many SDs 0 is away from 3.2:
(3.2 – 0) / 1.045 = 3.062 SDs.
Using a Z table, we find that 3.062 SDs below the mean only includes 0.11% of the data, so we have a confidence level of 99.89% that our true winrate is positive.

If we wanted to find the confidence level of an arbitrary interval, say (X, Y), we would do the same thing: use a Z table to figure out how much of the normal distribution falls below X and above Y; if 10% falls below X and 20% falls above Y, then (X, Y) is a 70% (100% - 10% - 20%) confidence level.

For those of you who plan to make similar calculations on a regular basis, I recommend familiarizing yourself with Excel’s functions NormInv and NormDist, which will allow you to skip using a Z table, and allow you to find values with greater precision.

end result

variance is 100times worse than anyone wants to admit, and often the gap between those that move up and those that don't has absolutely nothing to do with raw skill

Yea, as another example, consider a completely breakeven player with a standard deviation from the example I used in my OP (38.8439 ptBB/100), and assume he plays 50,000 hands. 10% of the time he will run at over 2.22 ptBB/100. At nl50 this amounts to profiting over $1113, and he'll probably move up to nl100 despite actually being a breakeven player. (Similarly, he'd run worse than -2.22 ptBB/100 over the same sample 10% of the time, probably busting his bankroll if he didn't move down).

Edit: The standard deviation I used in my original example is an actual standard deviation from my last 100K+ hands at nl200. Nits at FR have lower variance, but still probably have standard deviations over 25 ptBB/100.

That sounds pretty naive.

SD of 75ptBB/100 and winrate of 17.76ptBB/100 = lotso variance. yay!

using OP's example

identical player in skill

player A runs 2SD hot
player B runs 2SD cold

over 138k 1/2hands

player A made 29.2k
player B made 6.1k

player A is in way way better shape for moving up, even though he's the exact same skill as player B

we shrug off 2outers, but when the difference from 17.7k and 6.1k is a 2outer chance over a pretty decent sample, the variance is absurd

underestimating the luck factor in poker is a fatal mistake

holdem2000, thanks for this post...VERY good info

question: how many hands would be needed to have a 95% CI that someones wr is within a +/- 0.5 ptbb/100 of a target given their std dev/100 hands is 25 ptbb? 35ptbb?

i'm guessing this number is huge but wanted your math to verify how huge....thanks

also post the math plz if you could...thanks

So first off, a 95% CI will correspond to about 2 standard deviations for the whole sample around the winrate (more precisely, 1.96 SDs around the wr). So we need enough data that 1.96 SDs is only 0.5 ptbb/100, so we need 1.96 * SD = .5, so
SD = 0.5 / 1.96 = .2551 ptbb/100.

Taking our 100 hand SD to be 25 ptBB, and using the shortcut from my OP, we need then
.2551 = 25 / sqrt(N), where N is the number of hundreds of hands we played. Solving we get:
sqrt(N) = 25 / .2551 = 98.00
N = 98.00^2 = 9604

So with a 100 hand SD of 25ptbb, we need to play 960,400 hands to have that kind of CI.

Similar calculations lead to needing 1,882,300 hands for the same width CI if our 100 hand SD is 35ptbb.

Practically of course, our playing style and the game conditions change way before we ever get that kind of level of certainty. Incidentally if you were to just remember that a 95% CI is 2 SDs wide (and skip looking up the 1.96 value from a Z table or from Excel), you'd get that you need 1,000,000 hands in the first case and 1,960,000 hands in the second, not too dramatic of a change).

bump

as evident by holdems calculations. It takes over 1M hands before we begin to even narrow in on our true win rate

thx for the calculations!

the other tricky part is, that unless your name is fgators most people won't play a million hands with the same exact style, without trying to improve

just sit back, realize the longrun doesn't exist, and play your best every session and work on your game basically

I like.

I'm great. I suck. I'm great. I'm great. Wait, I suck, I suck, I suck. Let me consult PT. Hmmm. Really no way to know. Consult Zen master. Snatch pebble from hand. I am.

Great thread.

QFT

Agreed. Nice work Holdem & Pokerboy.
Mods - Please sticky.

I couldnt get my thick head around this maths until now.

+1

Isn't it likely that the curve representing a poker players win rate per 100 hands is not normally distributed. There are other distributions besides normal distributions, and the "skill" factor should mean that, on average, a skilled player who has the same normal distribution of starting hands and flops (normal distribution based on RnGs allocation of cards) will win more and lose less than a less skilled player. The card and hand distribution should be normal, but the amount of wins and losses might not be.

Is there a way to plot several winning and losing player PT results regarding win rate/100 distributions over 200K hands to see if the "curve" looks normal vs some other, skewed distribution curve? It is likely, I think, that the curves are skewed with the bad players losing more money more often with the same cards thand the good ones.

Nice work. I knew some math but this refreshed it a little. Seems i'm 99.3% a winning player at 3.6 +/- 1.5 ptBB/100 at 25NL (40k hands / MT ratio = 4)

Thanks for all the work Holdem, great thread.

Please Sticky.

Will need to read this a few times (and the rest) to take it in.

.......


Interesting view bottomset, I've always tried to approach poker in the "long term" view. Yet you believe there to be no long term game.

Has my mind been corrupted by poker books?

Absolutely true. There are facets of variance we often don't even consider. For instance when we run into set over set it's a cooler and we move on confident that it will happen in our favor and all will be even.

But what happens if we lose a full BI twice when we hit the wrong end of set over set but when it goes in our favor twice it's against some short stack? Same with AA vs KK.

For some of these kinds of variance the "long run" may be too long to even out.

No, there is a theoretical long term. It just may be longer than you play cards for.

if i didn't mess up the math, here are some simulations with a winrate of 3 and standard deviation of 35 over 50k hands:


100k hands: