I've just completed a series of hand analyses to look for evidence of bias in the deal at some major poker sites/networks.

Over 6 million hands were analysed from: PokerStars, Party Poker, Ongame Network, Merge Network, Entraction Network & Bodog. These sites comprise at least 85% of all online poker traffic according to Poker Scout.

The analysis compares the expected number of hands won to the actual number of hands won for flop all-ins. Some subsets of the whole sample are isolated so that results of these can be compared as well:

the caller is isolated to see whether there is any bias towards those who call the all-in on the flop.

hands with equity<50% are isolated to see if there is any bias towards the underdog in all-in situations on the flop.

I've also made some changes to take on board things that have been said about the previously run pre-flop tests, such as:

Having a clearer explanation of what the results mean and how to interpret actual deviation alongside standard deviation for each sample.

As suggested I've included probabilities of how often samples that are close to 2 standard deviations or more should occur due to variance alone.

Included is a margin for error of the final results (due to the Monte Carlo).

Included a discussion page that highlights possible weaknesses in the tests.

Thanks everyone for your input on the last tests and I welcome discussion of these tests too. I'd be grateful if you let me know if you spot any mistakes - it's easy to slip up occasionally.

See the whole test and results here.

Thanks for putting these up.

I see that Partypoker came out at +2.17 SD's which surely is a fail at the 95% confidence level isn't it?

Also Bodog was +1.94 which is about borderline.

5 out of the 6 came out with a + deviation (ie the underdog did better than expected). The one which came out negative was quite small sample of only 3674 hands.

If I lump all the data together from the 5 sites I come up with 76059 hands of which the best hand won 59442 against an expected 59831.8. This has an SD of 112.98 so you are 3.45 SD's adrift of the mean with probability 0.000295. Apologies if I have made a mistake in copying the data or in a formula - Can you check what you come up with?

I am quite surprised nobody has commented yet on the results.

Another thing: I can see the rational of applying a 4 or 5 sd rule as 'proof' if you are talking about an individual's HH's. After all, as has been pointed out before, among 100,000 players there will be some who have a particularly extreme sob story.

However with these analyses across many players we are talking presumably about a completely random sample with no cherry-picking involved . If that is the case I see no reason why a 95% confidence level should not be applied when talking about a pass/fail test for randomness. Anyone have any views on this?

True - Party Poker came out at +2.17 deviations from the mean. This should occur about 3% of the time or once in every 33 tests.

It doesn't meet the 95% confidence level but I don't think the 95% confidence level is very useful here. 1 in every 20 tests should fail at the 95% confidence level due to variance alone. I've run 18 tests already, so this would be nothing unusual - this is the only one that's outside the 95%.

Re: lumping the sites together. I'll check your figures and let you know if I get the same result. Is lumping the sites together good practice or not??? They have different methods of dealing etc.

Views on this anyone...

If there is nothing going on in reality to favour the underdog then the effect of lumping multiple sites together should be to reduce the overall SD. The fact that the results are reinforcing each other would seem to indicate that something funny is going on.

I don't see your argument about the 95% either. OK if you have done 20 separate independent tests of partypoker data and the results show a distribution about a mean of zero, with one outside the 95% level then OK. But I don't think you have 19 similar tests of postflop equity run on partypoker have you?

What you have is 5 tests which have a mean nowhere near zero, and a probability of the parent population having a mean of zero as a very small number.

I really hope too that some of the mathematicians can have a look at the results in detail and give their views.

Just done some calculations with all the sites together. For all the underdog hands in the sample I get:

Actual Deviation = +210.2
Standard deviation = 107.3

so +1.96 standard deviations from expectancy

Why does the chart say "heads-up, pre-flop" instead of "heads-up, flop"? Is that just a mistake?
http://www.ispokerrigged.com/flop-all-in_dataset.html

Here is how I came up with my result:

I added up all the expected won for the good hands then added the hands lost by the bad hands to get 59831.8. I did the same for the actual won to get 59442. This out of 76059 total hands.

The SD is sqrt(N*p(1-p)) = sqrt(76059*(59831.8/76059)*(1-59831.8/76059)) = 112.98
The actual deviation was 59442 - 59831.8 = -389.8
This is -3.45 SD's from the mean.

You can do the same for the underdog, which yields +3.45 SD's from the mean.

Can you explain where your 210.2 came from?

Well spotted - just a typo. Now corrected.


That is not how I worked it out. Bear in mind that the total number of hands in the whole sample includes some hands of equity=50% that are not either 'ahead' or 'behind'.

I got the 210.2 simply by adding the actual deviation (of all underdog hands) for each of the sites together.

Actual Deviation (underdog hands):

Bodog 76.2702
Merge -5.3444
Entraction 28.693
Ongame 6.9962
Party 90.1025
Stars 13.4744

Total 210.1919

Each site's actual deviation (of all underdog hands) was calculated by subtracting the actual deviation of the 'ahead hands' from the actual deviation of the 'behind hands'.

The actual deviation of 'ahead hands' and 'behind hands' was calculated by subtracting the expected number of hands won from the actual number of hands won.

Since the binomial distribution is non-perfect (i.e. a different probability for each hand), n*p(1-p) gives a slightly inaccurate result. So I've used: sqrt{sum[p(1-p)]} where p(1-p) is calculated individually for each hand to get an accurate result. Thanks to BruceZ's help in this thread for directing me to this method.

Ok, so from these tests we see that overall the underdog are favored to quite some degree. Do I need to remind you that these results are good for the pokersite rake-wise?

So if the site deals a random deal, this is a good result from them. They got lucky here.

Or perhaps they cheated and chose to get this good result, and we are left without proof anyway?

Found the problem - another typo. In the Party Poker, Ahead hands, Actual hands won box I'd put 5429.3 on the website and the number should be 5249.3 oops!

This was just a typo made when transferring that individual number to the website and doesn't affect any of the other results as they were copied straight from the original spreadsheet. This has now been corrected and all the numbers should tally - fingers crossed!

When I did my calculations I used my original spreadsheets which is why I our numbers didn't match. You can see a copy of the original spreadsheet by viewing the PDFs - the number is calculated by summing all the hands and is listed at 5249.2552 (it can be checked by working it out from all the hands listed if anyone has doubts).

Great to see the figures getting the fine tooth comb treatment

In these tests it was shown that the underdog is favoured slightly but the degree is very small really and well within the limits of variance.

In the pre-flop tests the favourite was favoured slightly, but again in a very small way.

I don't think that these results show that anything unusual is going on at all.

I found the error too!

OK I reworked using 5249.3, and now get an SD of 1.85.

I see what you are saying regarding the inaccuracy in just using the average p in the SD formula but this is not really a gamechanger (I got 112.98 and you get 107.3).This explains why you get 1.96 instead of 1.85.

I would not be so quick to say that 1.96 is well within the expected bounds of variance. The fact is that it falls outside the 5% confidence level, and I still see no reason not to take that as a criterion for pass/fail.

I think ideally your website should give the probability of the result in terms of SD's. Then people can decide themselves what confidence level they require. If your definition of fail is 5 SD's then you should state that that corresponds to a million to 1 or whatever. Personally I am not sure I want to play somewhere that only has a 3% probability of being random.

Cheers for all your good work anyway.

You don't understand what that number means.

It says absolutely nothing about the chance the site is random, only that this result or a worse one will happen 3% of the time *if* the site is random.

http://imgs.xkcd.com/comics/significant.png

There was only one test on green jelly beans causing acne in this cartoon as well.

But, if you do ~20 tests on *anything* there is a significant chance that *one* of them will fail a hypothesis test at the 5% CI. Obviously, the chance is 0.95^20 that all pass, so 36%. Given that its likely that one test fails by chance, which one it is is completely meaningless, hence the joke behind the cartoon. Party Poker is no more likely to be rigged on the basis of these results than some color of jelly bean is to cause acne.

Also, 2.17 s.ds doesn't fail a hypothesis test with a 95% confidence interval - not sure why you think it does. 95% CI goes from 2.5% -> 97.5% chance of a worse result (since we would be interested if the underdog lost more than expected as well).