Quote:
Originally Posted by bip!
The general question (I think what the poster is after) -
is people hear:
"X hour sample is meaningless!"
... then they actually think independently and realize 'well, it can't be meaningless... but I wonder how much meaning it has..'
And I don't think any ponderer's concern is seven digit accuracy of confidence.. they just want to know if being up 2000 bb in 150 hours gives you 1% confidence in being a long term winner or 70% confidence in being a long term winner.
So in my opinion, who gives a **** if poker statistical modeling might get an F on a master's thesis - it still is practical and functional and better than saying "it is impossible to calculate". It is impossible to be perfect, but very possible to be close IMO.
At about that hour in my results, I calculated my percent chance of being a winning player through some likely hack method... It was over 90% - which allowed me to confidently take shots in higher games. I will try to find the post...
(Now having an actual distribution of win rates would be awesome - and help a ton with analysis - but that data collection will never happen)
I'm with you 100%. All models are wrong, but some are useful... as they say.
My point is that a Stats-101 confidence interval is not the best way to get a feel for how likely it is that your win-rate accurately reflects your ability. The problem with such a CI, which is really super super super easy to calculate, is that it is too precise and the assumptions beneath it are really problematic for thinking about a game like poker. The paradigm is just all wrong and it's wrong in ways that will give people the wrong idea about their game.
The problem with a confidence interval is tells us how often the constructed interval, based on our data, is likely to contain the true population parameter (our true win rate). The confidence is about the interval, not the estimate per se. We'd rather be able to talk about the confidence we have in our point estimate, not the interval around it. Bayesian statistics is the best way to arrive at such estimates.
Anyway I think all that's unnecessary here. My point is that someone, in my opinion, will get a better idea of the likelihood in there win rate by plotting their win rate over time and empirically looking at its shape and pragmatically assessing the data.
Some things we want to look at:
* Mean: average win rate, easy. But can be distorted.
* Median: I would contend a better estimate of your win rate. The nature of winning big pots or losing huge ones can result in some godly sessions and horrid sessions which may reflect random luck/bad luck more than skill. Median will ignore these.
* Standard deviation: this quantifies the spread of our win rate. If this is large, we should have less confidence that we are a winner, especially over smaller samples.
* Skewness: This is really, really important. There are ways to calculate skewness statistics, but the easiest way is to just make a histogram or kernel density plot of your win rates across sessions. If there are long tails on the negative side of winning, enough to seriously pull the mean towards these losses, we might learn something more important than "I am a winner or loser." We might learn: I am a winner in the vast majority of sessions but I have a huge tilt problem. Another easy way is just to compare your mean to median win rate. If they are substantially different in one way or another, that means you have large outliers skewing your data.
* Sample size: duh.
Of course, the prudent will realize the above are precisely what's necessary to calculate confidence intervals. But my point is that a confidence interval is much less helpful than actually plotting out your rates and analyzing it, and coming to a conclusion like:
"I'm mostly a winner: I've left a winner in 2/3 of the last 100 sessions. But I have the potential to tilt and really blow it: my mean win rate is substantially lower than my median rate, and it's costing me. On the other hand, I haven't booked nearly as many large wins -- I tend to win a lot and get overconfident and pick bad spots. My standard deviation is also quite large -- more than a buy-in at my game! That means the average difference between any one session's win rate and my mean rate is more than a buy-in. I am a high variance player."
That's a data-informed way of giving a player an idea about how good you may bb. It's far better than saying: My average win rate is X, with standard deviation Y, so assuming Z, 1.96xSD/sqrt(n) +- win rate = the boundary I expect my win rate to be.
A last point is that when we use inferential statistics like confidence intervals, we try to estimate a true parameter based upon a sample. Poker is weird in that we actually always have access to our true win rate, so long as we keep good records. We don't have to sample. We have complete data.
Our concern is our tendencies and the confidence that these rates accurately reflect a long-term trend. Diagnostic analysis of sample size, mean, median, standard deviation, skewness etc. will give us a much better analysis.
People can go ahead and construct a CI around a win-rate and say "I have 95% confidence that my true win rate lies in this interval." The thing is -- we don't need to guess at our true win rate. It changes over time and can be calculated. Instead we need to understand what's lurking underneath the summary statistic.