What's wrong with PokerRoom.com's RNG? - Gambling and Probability

Two Plus Two Forums Other Topics Probability

What's wrong with PokerRoom.com's RNG?

Post Reply Subscribe

...

Page 1 of 2

1 2

Page 1 of 2

1 2

11-10-2008 , 03:18 PM

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi,
I just did a Chi-Square-Test with the sample of dealt hands given at
http://www.pokerroom.com/poker/poker...stats-by-card/.
I got a hint from this post:

Quote:

Originally Posted by McSeafield

(...)
This is an example where something could be incorrect - either the statistik because of unexplained card removal effects or Pokerroom has a serious RNG problem. One can prove this with a significance test.

Look at all poket pairs 55+, all Axs, all T9s+, QJo, KTo+ and ATo+. These hands are 244/1,326 = 18,4% of the best starting hands and appeared only 22,430,255 times. The expectation was 22,455,221 hands. There is a difference of 24.996 hands. One SD amounts SQRT(122.031.244 * 244/1326 * (1-(244/1326))) = 4,281. 24,996/4,281 = 5.84 SD. This is very huge and should not happen.

The biggest deviations have Queen-Combos: QTs(-4.49 SD), QQ(-3.52), AQo(-2.97), KQs(-3,18) and KJs(-3,63), 66(-3,35), AA(-2,08). All pocket pairs 55+ together > -5 SD.

I used the following expected frequencies:

Code:

Cards	observed	expected
AA	550,632	  552,178
KK	551,878	  552,178
QQ	549,570	  552,178
JJ	550,948	  552,178
TT	550,156	  552,178
99	552,062	  552,178
88	550,710	  552,178
77	553,492	  552,178
66	549,696	  552,178
55	550,840	  552,178
44	552,443	  552,178
33	551,586	  552,178
22	553,171	  552,178
AKo	1,106,047	  1,104,355
AKs	367,870	  368,118
...     ...             ...
43o	1,106,577     1,104,355
43s	368,525	  368,118
42o	1,101,359	  1,104,355
42s	369,977	  368,118
32o	1,103,272	  1,104,355
32s	369,182	  368,118

The total size of the sample is 122,031,244 hands.
The Results:

Probability, that the observed hands are dealt due to chance: P = 1.88157E-36, ok essentially P = 0.

Probability, that the observed pairs 22+ are dealt due to chance: P = 9.55772E-07, ok essentially P = 0, too.

Given the usual confidence level of p = 0.05, you can safely reject the hypothesis that the hands are dealt by a change process.

For comparison I made the same test with 91,118 drawn hands from one of my PockerTracker databases. The hands are not from PokerRoom.com, of course.

Probability, that the observed hands are dealt due to chance: P = 56.15%

Probability, that the observed pairs 22+ are dealt due to chance: P = 26.19%

Here everything seems to be all right.

I'm not an expert, maybe somebody can explain what's going on here.

Thanks,

/SunKidDance

Quote

11-10-2008 , 10:54 PM

DrVanNostrin

Pooh-Bah

Join Date: Sep 2005 Posts: 4,402

I'm gonna go out on a limb and say you've done the chi-squared test incorrectly. I certainly could be wrong here. Post all the data and I'll take a look at it.

If you have n different starting hands and cards are truely random, then the test statistic will follow a chi-squared distribution with n-1 degrees of freedom.

The chi-squared statistic is equal to the sum of [(E-A)^2]/E, where E is expected and A is actual.

Quote

11-10-2008 , 11:31 PM

Sherman

Carpal \'Tunnel

Join Date: Jun 2005 Posts: 7,766

Quote:

Originally Posted by DrVanNostrin

I agree here, and I am usually pretty skeptical of claims that internet poker websites are broken.

However, if you just do the chi-square on the 21 (out of 169 total hands) he has displayed here you get a chi-square of 98.977 on 20 df. This has a p of 1.92E-12 alone.

Or if you just look at pocket pairs, you get a chi-square of 69.64 on 12 df. I get a p of 3.73E-10.

I'm more likely to question the collection of the observed data than the math.

Sherman

Quote

11-10-2008 , 11:41 PM

DrVanNostrin

Pooh-Bah

Join Date: Sep 2005 Posts: 4,402

If you only look at 21 hands, you need to use a chi-squared with 21 degrees of freedom, not 20. But I don't think this will make much of a difference.

Quote

11-11-2008 , 02:51 AM

Sherman

Carpal \'Tunnel

Join Date: Jun 2005 Posts: 7,766

Quote:

Originally Posted by DrVanNostrin

If you only look at 21 hands, you need to use a chi-squared with 21 degrees of freedom, not 20. But I don't think this will make much of a difference.

Huh? What am I missing here? First, the degrees of freedom for a chi-square goodness of it test (which is what we have here) is K-1 where K is the number of groups (in this case hands). You just said a few minutes ago that with N total hands the DF = N-1. So when I say N = 21, you say DF = 21? That makes no sense.

But for what it's worth, no it does not make any difference if you use 20 or 21 degrees of freedom.

Sherman

Quote

11-11-2008 , 02:02 PM

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi,

Quote:

Originally Posted by DrVanNostrin

I'm gonna go out on a limb and say you've done the chi-squared test incorrectly. I certainly could be wrong here. Post all the data and I'll take a look at it.

Sure, here is the complete data (169 different hand combinations as given in the PokerRoom-link:

Code:

Cards	observed	expected
AA	550,632	552,178
KK	551,878	552,178
QQ	549,570	552,178
JJ	550,948	552,178
TT	550,156	552,178
99	552,062	552,178
88	550,710	552,178
77	553,492	552,178
66	549,696	552,178
55	550,840	552,178
44	552,443	552,178
33	551,586	552,178
22	553,171	552,178
AKo	1,106,047	1,104,355
AKs	367,870	368,118
AQo	1,101,249	1,104,355
AQs	368,178	368,118
AJo	1,103,946	1,104,355
AJs	367,811	368,118
ATo	1,105,376	1,104,355
ATs	367,393	368,118
A9o	1,105,684	1,104,355
A9s	368,279	368,118
A8o	1,100,445	1,104,355
A8s	368,982	368,118
A7o	1,104,965	1,104,355
A7s	369,231	368,118
A6o	1,105,125	1,104,355
A6s	366,998	368,118
A5o	1,104,643	1,104,355
A5s	367,900	368,118
A4o	1,104,763	1,104,355
A4s	367,553	368,118
A3o	1,105,722	1,104,355
A3s	367,269	368,118
A2o	1,106,519	1,104,355
A2s	366,466	368,118
KQo	1,103,231	1,104,355
KQs	366,191	368,118
KJo	1,105,604	1,104,355
KJs	365,921	368,118
KTo	1,103,705	1,104,355
KTs	368,086	368,118
K9o	1,103,920	1,104,355
K9s	367,736	368,118
K8o	1,106,439	1,104,355
K8s	369,893	368,118
K7o	1,101,741	1,104,355
K7s	367,647	368,118
K6o	1,103,401	1,104,355
K6s	366,407	368,118
K5o	1,105,669	1,104,355
K5s	368,807	368,118
K4o	1,104,957	1,104,355
K4s	368,061	368,118
K3o	1,104,211	1,104,355
K3s	368,321	368,118
K2o	1,106,898	1,104,355
K2s	368,737	368,118
QJo	1,102,901	1,104,355
QJs	368,213	368,118
QTo	1,106,012	1,104,355
QTs	365,398	368,118
Q9o	1,107,991	1,104,355
Q9s	367,923	368,118
Q8o	1,106,395	1,104,355
Q8s	367,657	368,118
Q7o	1,104,331	1,104,355
Q7s	367,512	368,118
Q6o	1,105,012	1,104,355
Q6s	368,294	368,118
Q5o	1,106,053	1,104,355
Q5s	369,538	368,118
Q4o	1,103,742	1,104,355
Q4s	368,471	368,118
Q3o	1,106,081	1,104,355
Q3s	367,390	368,118
Q2o	1,104,650	1,104,355
Q2s	368,355	368,118
JTo	1,102,233	1,104,355
JTs	367,811	368,118
J9o	1,102,303	1,104,355
J9s	369,223	368,118
J8o	1,104,552	1,104,355
J8s	367,615	368,118
J7o	1,105,297	1,104,355
J7s	364,541	368,118
J6o	1,104,704	1,104,355
J6s	366,958	368,118
J5o	1,105,844	1,104,355
J5s	368,354	368,118
J4o	1,106,654	1,104,355
J4s	366,906	368,118
J3o	1,104,433	1,104,355
J3s	368,616	368,118
J2o	1,103,759	1,104,355
J2s	367,858	368,118
T9o	1,103,441	1,104,355
T9s	367,750	368,118
T8o	1,107,310	1,104,355
T8s	366,732	368,118
T7o	1,103,171	1,104,355
T7s	367,201	368,118
T6o	1,101,165	1,104,355
T6s	368,322	368,118
T5o	1,107,072	1,104,355
T5s	368,030	368,118
T4o	1,106,174	1,104,355
T4s	370,150	368,118
T3o	1,102,033	1,104,355
T3s	369,188	368,118
T2o	1,105,503	1,104,355
T2s	369,195	368,118
98o	1,103,082	1,104,355
98s	368,190	368,118
97o	1,104,506	1,104,355
97s	369,494	368,118
96o	1,105,092	1,104,355
96s	369,159	368,118
95o	1,102,769	1,104,355
95s	368,015	368,118
94o	1,105,939	1,104,355
94s	367,617	368,118
93o	1,104,310	1,104,355
93s	368,278	368,118
92o	1,107,579	1,104,355
92s	367,488	368,118
87o	1,103,007	1,104,355
87s	367,787	368,118
86o	1,105,837	1,104,355
86s	367,805	368,118
85o	1,106,745	1,104,355
85s	367,456	368,118
84o	1,104,856	1,104,355
84s	368,694	368,118
83o	1,106,532	1,104,355
83s	368,814	368,118
82o	1,101,727	1,104,355
82s	368,039	368,118
76o	1,105,164	1,104,355
76s	367,110	368,118
75o	1,105,498	1,104,355
75s	369,303	368,118
74o	1,101,551	1,104,355
74s	369,655	368,118
73o	1,105,302	1,104,355
73s	368,640	368,118
72o	1,104,285	1,104,355
72s	368,039	368,118
65o	1,104,700	1,104,355
65s	367,986	368,118
64o	1,101,489	1,104,355
64s	369,101	368,118
63o	1,103,080	1,104,355
63s	365,732	368,118
62o	1,107,570	1,104,355
62s	367,206	368,118
54o	1,104,529	1,104,355
54s	367,333	368,118
53o	1,105,251	1,104,355
53s	366,243	368,118
52o	1,107,694	1,104,355
52s	367,876	368,118
43o	1,106,577	1,104,355
43s	368,525	368,118
42o	1,101,359	1,104,355
42s	369,977	368,118
32o	1,103,272	1,104,355
32s	369,182	368,118

Quote:

Originally Posted by DrVanNostrin

If you have n different starting hands and cards are truely random, then the test statistic will follow a chi-squared distribution with n-1 degrees of freedom.

Because the system is fully specified, you are right. The degrees of freedom are 169-1 = 168 for the PokerRoom sample.

Quote:

Originally Posted by DrVanNostrin

The chi-squared statistic is equal to the sum of [(E-A)^2]/E, where E is expected and A is actual.

Yes, the chi-squared statistic for the PokerRoom hands (169) is 512.02.
For the pairs 22+ it's 50.94.
So let's look up p with the Excel function CHIVERT(chi-square statistic, degrees of freedom):
p = CHIVERT(512.02,168) = 1.88157E-36 for all hands
p = CHIVERT(50.94,12) = 9.55772E-07 for the pairs.

Sorry, I can't see an error in the calculation, but please check it

, I'm curious.

Regards,

/SunKidDance

Quote

11-11-2008 , 02:23 PM

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi Sherman,

Quote:

Originally Posted by Sherman

I agree here, and I am usually pretty skeptical of claims that internet poker websites are broken.

Me, too. I just looked at the data, that's all.

Quote:

Originally Posted by Sherman

However, if you just do the chi-square on the 21 (out of 169 total hands) he has displayed here you get a chi-square of 98.977 on 20 df. This has a p of 1.92E-12 alone.

Or if you just look at pocket pairs, you get a chi-square of 69.64 on 12 df. I get a p of 3.73E-10.

I'm more likely to question the collection of the observed data than the math.

Maybe, but this would be the problem of PokerRoom.com. If they published wrong data, they have something to explain, I suppose...

Regards,

/SunKidDance

Quote

11-13-2008 , 02:37 PM

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi DrVanNostrin,

I uploaded the Excel sheet to Google docs:
http://spreadsheets.google.com/ccc?k...xDqr5VnVhEqUbg

So, it's easy to check the computations.
So, go on and tell us your results.

Thanks & Regards,

/SunKidDance

Quote

11-13-2008 , 08:30 PM

Sir_Weed

newbie

Join Date: Nov 2008 Posts: 18

I did the test as well and got the same results as you. I couldn't find a mistake in your calculations. I'm pretty skeptical about PokerRoom being rigged as well so I'd like to think they published wrong data... But this is weird, how is it possible to publish wrong data like that? I hope our calculations are wrong

Edit: btw, I did the calculations on my own before you posted the link to google docs sheet (it's easier to overlook a mistake when checking someone's else calculations than to do the same mistake when calculating on your own).

Quote

11-14-2008 , 12:31 AM

#10

DrVanNostrin

Pooh-Bah

Join Date: Sep 2005 Posts: 4,402

Quote:

Originally Posted by Sir_Weed

I did the test as well and got the same results as you.

this

If the data is correct this is pretty ****ed up.

BTW, I'm still a little confused about the test of only 21 hands. But I'm pretty sure it's not right to use an incomplete set. You lose degrees of freedom because 169th value is not free. You know what it is based on the first 168 hands. When you only use 21 hands the 21st value cannot be determined using the first 20 results.

Quote

11-14-2008 , 03:21 AM

#11

Sir_Weed

newbie

Join Date: Nov 2008 Posts: 18

I'm not completely sure but I think Chi square test is used on events that have a total probability of 1, in other words you can not use it on incomplete sets.

Quote

11-15-2008 , 05:47 AM

#12

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi Sir_Weed,

thanks for checking. I'm glad the computations are technically correct.

Regards,

/SunKidDance

Quote

11-15-2008 , 05:54 AM

#13

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi DrVanNostrin,

Quote:

Originally Posted by DrVanNostrin

BTW, I'm still a little confused about the test of only 21 hands. But I'm pretty sure it's not right to use an incomplete set. You lose degrees of freedom because 169th value is not free. You know what it is based on the first 168 hands. When you only use 21 hands the 21st value cannot be determined using the first 20 results.

You are right. You can make the chi square test on a subsample only if you draw the subsample from the sample by a chance process, otherwise the subsample is biased and did'nt reflect a chance process. So, ignore the calculations for the pairs sub sample.

Quote:

Originally Posted by DrVanNostrin

If the data is correct this is pretty ****ed up.

Indeed.

Regards,

/SunKidDance

Quote

11-15-2008 , 11:31 AM

#14

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi,

I see three possibilities here to explain the result:

Card removal effects
The expected values as given above for the calculation of chi-square statistic are computed for drawing with replacement. In reality, the drawing is without replacement.
I don't know for sure whether this is a problem or not. Maybe somebody here can explain it further or give a quantitative estimation of the card removal effect.
The counting of starting hands on PokerRoom.com is flawed
Ok, then the results of the chi-square test didn't reflect the random dealing process. But if they can't count the hands properly...
The RNG of PokerRoom.com is flawed
Nothing to comment here.

One has to show that 1. is not a problem, i.e. the card removal effects can either be ignored or accounted for (how?), to consider 2. and 3.

Regards,

/SunKidDance

Quote

11-15-2008 , 01:06 PM

#15

Sir_Weed

newbie

Join Date: Nov 2008 Posts: 18

Uhm... I think the expected values above are computed without replacement.

Take for example expected value for a pocket pair: 4/52 * 3/51 * 122,031,244 = 552,177.5747.

Or... did I misunderstand your 1. possibility?

Quote

11-16-2008 , 01:44 PM

#16

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi Sir_Weed,

Quote:

Originally Posted by Sir_Weed

Yes, I think so. Look at the drawing process in Hold'em for ten players:
The first player gets the first card, p = 1/52, the second player gets the first card, p = 1/51,..., the tenth player gets the first card, p = 1/43.
Then the first player gets the second card, p = 1/42,..., the tenth player gets the second card, p = 1/33. Dealing is done.
The process is different and maybe also the expected values. But I don't know the impact on the expected values

.

Regards,

/SunKidDance

Quote

11-16-2008 , 03:18 PM

#17

Sherman

Carpal \'Tunnel

Join Date: Jun 2005 Posts: 7,766

Quote:

Originally Posted by SunKidDance

Hi Sir_Weed,

Yes, I think so. Look at the drawing process in Hold'em for ten players:
The first player gets the first card, p = 1/52, the second player gets the first card, p = 1/51,..., the tenth player gets the first card, p = 1/43.
Then the first player gets the second card, p = 1/42,..., the tenth player gets the second card, p = 1/33. Dealing is done.
The process is different and maybe also the expected values. But I don't know the impact on the expected values

.

Regards,

/SunKidDance

No. The logic here is wrong. You have no idea what cards were dealt to those players so you have no way of assuming they have made it "more difficult" for you to draw pair. The expected values you figured originally are correct.

Sherman

Quote

11-16-2008 , 07:14 PM

#18

Karganeth

old hand

Join Date: Jun 2008 Posts: 1,305

Quote:

Originally Posted by SunKidDance

Hi,

I see three possibilities here to explain the result:

Card removal effects
The expected values as given above for the calculation of chi-square statistic are computed for drawing with replacement. In reality, the drawing is without replacement.
I don't know for sure whether this is a problem or not. Maybe somebody here can explain it further or give a quantitative estimation of the card removal effect.
The counting of starting hands on PokerRoom.com is flawed
Ok, then the results of the chi-square test didn't reflect the random dealing process. But if they can't count the hands properly...
The RNG of PokerRoom.com is flawed
Nothing to comment here.

One has to show that 1. is not a problem, i.e. the card removal effects can either be ignored or accounted for (how?), to consider 2. and 3.

Regards,

/SunKidDance

Another explanation which you missed is that it the RNG is working properly and the calcualtions and card counting process are both correct. Just because it's unlikely that the cards came out that way without being biased doesn't mean it's impossible.

Quote

11-16-2008 , 08:13 PM

#19

undercheck

enthusiast

Join Date: Jul 2005 Posts: 57

Yep, card removal does not affect the expected values.

However, the observations are not independent, as the data contain starting hands of all the players from each hand. This may have an impact, as chi-square test assumes said independence.

Quote

11-17-2008 , 03:30 PM

#20

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi Karganeth,

Quote:

Originally Posted by Karganeth

given your assumptions the p-value is 1.88157E-36(!). Technically, the event is not impossible, but in reality it is. So I think your explanation is not an explanation grounded in reality.

Regards,

/SunKidDance

Quote

11-17-2008 , 03:38 PM

#21

SunKidDance

enthusiast

Join Date: Jun 2007 Posts: 82

Hi undercheck,

Quote:

Originally Posted by undercheck

Yep, card removal does not affect the expected values.

Ok, I want to believe

and I will check it with a simulation comparing both drawing processes.

Quote:

Originally Posted by undercheck

However, the observations are not independent, as the data contain starting hands of all the players from each hand. This may have an impact, as chi-square test assumes said independence.

True, thanks for this remark. The question is now, how big is the impact? So big that you can ignore it with 91,118 hands but can't ignore it with 122,031,244 hands?[/QUOTE]

Regards,

/SunKidDance

Quote

11-17-2008 , 05:09 PM

#22

Sherman

Carpal \'Tunnel

Join Date: Jun 2005 Posts: 7,766

All this talk of non-independence and different drawing processes is just wrong. The original analyses are correct. There is something wrong with Poker room's RNG, or the data they published is wrong, or my/our understanding of how they got the data is wrong, or everything is right and this event is extremely extremely rare.

Sherman

Quote

11-17-2008 , 05:18 PM

#23

McSeafield

Carpal \'Tunnel

Join Date: Jun 2007 Posts: 8,623

When I first looked at this data my selection criteria was "all starting hands with +EV". I did'nt search for flawed data. From my point of view we have here no selection bias.

If somebody is interested to find an explanation I suggest to contact the PokerRoom.com support and let the cardroom manager explain these differences. I think this would be best solution.

Last edited by McSeafield; 11-17-2008 at 05:25 PM.

Quote

11-18-2008 , 12:35 AM

#24

DarkMagus

Carpal \'Tunnel

Join Date: Jul 2007 Posts: 8,391

Quote:

Originally Posted by Sherman

I can understand that the expected values would be unaffected, but wouldn't the variances be different? For example on hands where an AA was dealt, it is significantly less likely than expected for the other hands to contain aces. I would expect this to cause a higher variance in the numbers than if the hands were dealt independently. Is the change in variance negligible, or am I missing something?

Quote

11-18-2008 , 01:31 PM

#25

wiggum

grinder

Join Date: Jul 2007 Posts: 430

Quote:

Originally Posted by DarkMagus

You are right. If I understand correctly how the data was collected, this is the aggregate data from all hands dealt to all players. Once a hand is dealt it affects all the other hand combination possible at the table. Each component of the chi-squared sum needs to be independent, so I can't see how this can be the case.

If on the other hand this data was collected from say a single person or position at the table (e.g. always the BB) then I think the analysis would hold. This is probably why your database of 93k cards shows no evidence of lack of randomness.

Quote

Page 1 of 2

First

1 2

Last

Post Reply Subscribe

...

Page 1 of 2

First

1 2

Last