You can play your A game, run bad, and then tilt. It isn't just C game --> bad results. You can also play your C game, run great, and still win.

If your results have a certain amount of bad outcome over time and that causes you tilt and then fall off a cliff by becoming a spewbot that may or may not factor in to your average. If this sort of thing happens a good number of times in your sample, maybe. If there is an event horizon beyond which you self-sabotage and go broke, it is probably rare if you're still playing. Also, you can hit the horizon of playbad, run really good and recover. If these are once every 50K or 100K hand phenomenon, most people have a really small sample. Again, you're breaking the rules on the math you're trying to use by not being random and uncorrelated.

Also, think about games steadily getting worse over time. You can't average that out. In fact, if you have a large hands in your database from 2005 playing on Paradise and Party, those hands represent games that no longer exist and skew your sample. Someone who plays huge samples now doesn't care, but as those hands are an appreciable part of your total they skew your results.

If all the guys that have a massive downswing would stop playing, wouldn't that skew the distribution to the positive side?
If games get tougher couldn't one also get better?
Effects can even out, you see...

I do get your concerns and I do agree that winrate is not some static thing. Your immediate winrate is a function of the table your sitting at and the state of mind you are in. You can sit down at a table and be +5BB/100, sit at another and be -5BB/100 at the same moment in your poker career.

However, your overall results after a lifetime of playing is a Gaussian distribution and the small term effects even out. That is what I mean in that they obey the CLT. Cranberry Tea actually brought out an extremely good point, but bar any effects like UGIEA, Black Friday, personal things like you moving, etc. the distribution of your results are Gaussian, so you are allowed to apply Gaussian statistics to it.

Is it unhelpful to question whether you're a 3BB/100 winner after 10k hands? Of course it is! Especially if it's your first 10k hands, and not just a stretch you played between last Tuesday and today.

However, I do find the statistical tools extremely helpful in a lot of situations - sorry, I think in terms of likelihoods rather than states of well-being.

I was just trying to address the one concern raised by the poster above (that bootstrap sampling might be required b/c the win rate may not be normally distributed.)

I understand these other concerns. I'm a trained statistician who has used sophisticated statistical tools in the real world for years. I'm well aware of the limits to the models.

Yeah, I'm with Bella on this one. After a few 100k runs all this A/B/C game stuff gets "smeared out".

I know people will disagree, but I also think there is some value in smaller sample sizes. If you've got some guy who plays 20-30k hands and runs 5BB/100, we can be somewhat confident that he's beating the game (i.e., about 90-95% confidence). All depends on how much certainty you insist on having.

And you can't take LHE results and predict NLHE success, and you can't mix stud and Omaha, and you can't just take an online NL100 winrate and use it for live $5-10. I think we all understand this.

(Whether this mix of game difficulty is actually skewing the shape of the distribution or just moving the mean is an interesting theoretical question that i'll have to defer to bellatrix, MApoker, and the other experts. I'm curious.)

I'm more concerned with practical use of winrate. I played 380 hours in a certain live O8 game in 2011-12. I also took about 100 hourly measurements to get a preliminary guess at SD. Sometimes that game had more loose-passive 80-yos than other times. Sometimes some solid 2+2ers were in the game, but not always. Sometimes the decent regs had been drinking, other times not. Every game situation is different, and yet i'm pretty confident my "true" winrate against a typical lineup for that game falls in the 95% CI i constructed.

Did i ever play against an absolutely typical lineup? Of course not -- but i can use my judgment to say, "The lineup looks a little tough tonight, so maybe instead of $20/hour, this game is worth $10/hour to me."

I think this makes sense, provided you exercise a bit of caution/take the results with a small grain of salt.

Remember, the primary use of these statistical tests is to rule out chance variation as the reason for the positive win rate. In these games, the chance variation generated purely by the cards is enormous compared to the slight differences in win rates that are generated by different players, etc.

If you have a win rate that is firmly 2+ SE's above zero over a wide range of playing conditions, I think you can be fairly confident that you're a winner on average, over this range of playing conditions. That doesn't mean you necessarily have a winning rate at any specific table (e.g. the night when all the sharks drop in while the fish stay home), but only over the range of games you typically play.

If something about the games you typically play changes over time (e.g. online now versus online in 2005), then yeah, your test is no longer applicable.

I have a much bigger problem with people who apply statistical inference tests when there isn't even a reason to believe there's any stochastic structure whatsoever -- e.g. data that are not generated from any systematic sampling process. People do this all the time, and it drives me nuts. At least here, we have a logical source for randomness (the cards).

You should understand the actual reason to do this stuff is to determine how much money you need to keep on hand to avoid going bust as a professional blackjack player. Poker pros adopted the idea for their own BRM.

Bellatrix convinced me to do fewer hands in session reviews -- basically, you can tell a ton about a player in a very few hands. I'd guess in a 100 hands of video review with a good player, you could know if you're a winner on the technical side. Tilt? Harder. Searching for one sided CI's to "prove" you're a winner is mostly about ego. Mostly, we get people who've run hot who self-select and run those CI's. There is a huge results bias in forum posting... and playing poker for a long time.

OK, but that's not what the OP is asking; rather he asked:

...and he proceeded to ask questions about how to calculate the standard error and confidence intervals. Those statistical tools (standard errors, confidence intervals, inference more generally) are not designed for BRM purposes. For one thing, they depend on how many hands you have in your data -- the greater your N, the smaller your SE and confidence intervals, whereas your BRM requirements should not depend on your N.

I'm not sure how you can determine this irrespective of your opponents. What if you're at a table full of people who play just as well as you do?

I suppose it may be for some people, but it's important to me for non-ego purposes. It's helping me to determine when/if I'm ready to level up, or whether a run of losses may be due to something non-random.

Also, I play live; I can't make videos of my play for review.

Table selection is a skill. Seat selection is a skill. Often in session reviews (which I recommend for everyone, see Dragon's post in micros), my comment is "you played great, but why did you even sit here?" Stats don't cover it because it really is about volume. Putting in volume in bad spots is bad. The correct play in some spots is to play less. BigBadBabar taught me this stuff, and helped me regroup after a 500+ BB downer. Before that, my one sided CI probably said I was a > 4BB/100 winner. Having played for income in mid-stakes online games, this stuff is from experience of having done it wrong personally.

Live, it is harder. You don't see hands. Online, I'd guess Bellatrix or BB could tell you in 50-100 hands if you stand to be a winner if you played in games they're familiar with. Game/seat selection is part of being a good long-term winner. If you're sitting in horrible spots, you are either playing for RB or just not winning. Live? Games are so much softer, still knowing when to play matters.

Well, that is the really true question in this thread, imo.
And one that can't be answered easily.
It probably is different for different people. Some people will be comforted if you tell them:
"Within 2-sigma, your results will lie in the within a 1BB/100 interval after 75k hands"
some people won't.

Some people will question the underlying assumptions going into that calculations, some people will just use it without knowing what it really means.

Sorry, joke of course, but I had to put it there:
"If you can't spot the sucker after half an hour at the table, then you are the sucker!"