Very interesting read. Solid thread

Noam is still a PhD student at CMU now in addition to working at Facebook. He has not defended his CMU dissertation yet.

It was break even, just as one can expect. It is a very mirror like curve. Too much variance to take the end result as final.

I agree that they are pretty agressive in their claims/marketing their results, but the variance reduction techniques they used were discovered independently and proven to be unbiased and usefull. So saying they didnt have big enought sample, or that their winrate estimates are wrong is idiotic. Prove them wrong if you are capable of doing so.

+1

just because we don't know what the variance reduction methods were does not mean that they are invalid

as another poster said, the ai can calculate the ev of its range at showdowns and use that instead of the ev of the hand that it actually had. sure it doesn't account for blockers, but it would also be possible to assign an arbitrary worst case scenario range (in terms of blockers) and then weight its own range based on that. This approach seems too conservative and crude, but we don't know what exactly they did. It's just clear that massive variance reduction does exist.

Using the Variance calculator (from here https://www.primedope.com/poker-variance-calculator/)the hypothesis that their variance reduction technique is correctly applied can be rejected.

Probability of running below observed win rate (-7.00 BB/100) over 10000 hands with a true win rate of 40.00 BB/100 is 0.0001%

Even if they mean small blind then the hypothesis can also be rejected as p<0.5%

Probability of running below observed win rate (-7.00 BB/100) over 10000 hands with a true win rate of 20.00 BB/100 is 0.3467%

There is no need. It finished with -7BB/100, that can be interpreted only one way.

Doesn't six humans, everyone in different positions, play against 5 bots and then compare it to the bot playing against itself in corresponding positions & same holecards, give pretty accurate results of true winrate? Is this the variance reduction method they used?

Having Chris Ferguson as a benchmark for team humanity seems wrong though.

where are you pulling 40bb and 20bb/100 from? the researchers did not claim those winrates

You're right. I just saw 40bb all over the thread and assumed it's what was claimed.

Apparently the claimed winrate is 4.8bb/100 hands, the observed winrate is -7.09bb/100 and depending on the std dev the likelihood of this happening is something between 5 and 20% so not impossible at all.

We know what the techniques are, they were published years ago (just google AIVAT). One of the original creators of this family of techniques even contributed to the thread.

To apply that you need to know full strategy with ranges at every point. You also have to implement it correctly. If they had ranges at any point in play then they could calculate exploitability (I won't go into technical details but it is trivial to approximate using monte carlo runs when you calculate range for one player and strategy for singular hands for other players). They claimed they can't calculate the exploitability though.

Well yeah, it's easy to make mistakes there and I wouldn't trust it's implemented correctly seeing how they already once played without blockers being included (which frankly is such a fundamental part of correct implementation that you can just assume something that doesn't include those is full of other errors). Looking at some random spews posted in this thread it looks like blockers are still a problem. If you look at precisely solved poker spots you will see that bluffs are never random hands but always chosen based on blockers. It's very unlikely that any decently close to equilibrium strategy contains random spews.

They are currently answering questions regarding this btw, not sure for how much longer:
https://www.reddit.com/r/MachineLear...omas_sandholm/

"I think we’re now done with poker. Going beyond two players was the last major AI challenge in poker"

Linus lost on purpose to stop further research in poker AI, what a genius

https://science.sciencemag.org/conte...cience.aay2400

Then Noam in AMA
lol wtf why can't they just be straight up and honest. One moment Linus is b/e vs bot+4 humans, another he's b/e vs 5 bots. Hard to take them seriously when they can't get a crucial piece of information like this correct.

it's sort of random on some boards though, like monotone boards or 4 straight, straight on the board, also sometimes in certain cases it's more geared toward bottom of range rather than blockers

vast majority of the time it is blockers though

I think that was referring to a separate thing. There was the 6max game featuring Pluribus and 5 humans, which was the subject of the paper, and another 6max game featuring 5 Pluribus and 1 human which happened after. The human winrates being cited, such as Linus' -0.5bb/100, are referring to the latter.

"We played 5 copies of the same bot vs Linus, so it doesn’t really make sense to look at the win rate of each bot individually. Being the same bot, they should all have the same win rate, and any difference would just be due to variance. (This experiment involving Linus didn't finish until after the final version of the Science paper was submitted, so it doesn't appear in the Science paper, only the blog post.)"

Here's the blog post that discusses the 5 Pluribus vs human setup

https://ai.facebook.com/blog/pluribu...-player-poker/

thx for posting the reddit link

More videos from youtube:

Isn't this very easy to skew in favour of the bot, by just applying more accurately calculated strategy when running the AIVAT technique, compared to what strat was actually played in game?

While I agree with bolded, according to the paper they did both scenarios you describe. bot x 5 vs 1 human and 5 humans vs 1 bot

Yes but they didn't say Linus did 1H+5AI in the paper and Noam acknowledged this and edited his response presumably after reading my post.

Hand 3 wtff

The variance reduction methods. Probably doesn't mean playing low variance, so I expect it to mean the true EV or so. The trackers show the all-in EV only and there are reasons for that. But if one can see all the cards, that's different.

https://youtu.be/5bcbqCqosAs?t=107

is this fold standard?, its just a 25% bet ott