completely disgraceful

I'd love to hear Sam's thoughts on this lol. Isn't he a two-time supernova elite? How could this statement pass through his bs filter???

Had Claudico been +9bb/100 rather than -9bb/100... I wonder what tune the professor would be singing

Also, I'm guessing that $170 million bet included uncalled bets. Wouldn't Claudico's many overbets that rarely get called massively increased the "pot" compared to the amount won even if the humans calling strategy soundly beat them? Perhaps those bets increased the standard deviation over normal human matches?

But yeah, complete bs that he didn't back it up with that actual statistics rather than, "look, here's a big number and here's an irrelevant small number so its a tie."

It's not too surprising after seeing him talk on stream for 2 weeks. He talks about Claudico like a beloved son. I'm not sure what's worse though, calling the end result a tie or when he told somebody that the bot was in 4th place out of 5.

The Prof probably doesn't like being told he is wrong if his interpretation of this event is any indication.

The best aspect of this event for me was the insight it gave me into the minds of academics. I will now no longer trust and most certainly will verify what "experts" say.

Won't the equity chops and mirrored hands be reflected in actual St Dev observed in 80K hands without need for adjustment?

a lot of low and midstakes players with near 170 millions wagered would love to have a statistical tie in their bank account

and once again people turn away from their ego problem to a professor's ego problem.

Doug said that he beats the top regs he plays for more than 9bb and that claudico is extremely good.
This bot is still better than every single person that posted in this thread besides bjorn/doug/dong

whats your point? everything said on this page is still true, this wasnt a tie it was a very clear victory for humans.

Nobody said the bot was bad.

epic, quoted for posterity

lol@tie

LOL@ tie

I really enjoyed these matches over the last weeks,it was very entertaining and they did a great job representing the poker community (especially Doug).

Look, we all heard clearly from Doug's statement that Claudico was a strong opponent and that he beats the top regs for more than the winrate against Claudico. No one is doubting Claudico's strength.

What is crazy and totally illogical is the Professor's means of evaluating the match. 9bb/100 tells us (according to Doug) that Claudico is a strong opponent. However, it is nowhere statistically insignificant- just non-conclusive. Look at the way they evaluated the results: looking at the total amount of money wagered, saying that the money won by the humans was a small fraction of that. Anyone who's played a decent amount of poker should understand how stupid that is.

As someone posted, consider the headline the Professor would write if Claudico was up 9bb/100. I'm sure it would not be statistical tie. A much better headline would be that the humans beat Claudico in a close match that demonstrated an AI can play at a world-class level in heads up no limit.

if it's a statistical tie, hopefully we will see some real money matches between top players and claudico with the professor backing claudico

Hey all,

I'm one of the Claudico developers and I'm sitting here chatting with the pros and Sam now. I thought I'd clear up some confusion about the statistics. We calculated the 95% confidence interval based on the 80,000 mirrored hands that were played and it was +/- 10.35bb/100. The pros won by 9.16bb/100. That's a pretty strong lead, but the result is not statistically significant at 95%.

We discussed this with the pros before we made the announcement and we all were pretty satisfied with how things were phrased. The title says that the pros finished ahead in chips, but the subheader says it was not a statistically significant result.

good segment. look forward to seeing the longer one.

That's great and a fine analysis, but that's not how it was remotely phrased in the article. You could have said it was statistically insignificant with that analysis attached, but what's actually written is really horrible:

"The "pot" won by the human poker pros—$732,713—sounds very impressive. But during the two week "Brains vs. A.I." challenge, $170 million was bet.

The winnings amount to less than one-half of 1 percent of the total pot—a statistically tiny amount."

With 80k hands, the observed variance probably should be used, rather than just assuming it's the same as the variance of human play. Same argument for directly including the effects of duplicate and equity chops, rather than just assuming it's around 25% for duplicate and ...?% for all-in equity. Has anyone been crazy enough to transcribe all of the hands from stream?

I'd have guessed a standard deviation of around 16-18 big blinds/hand (sorry about units!) with all-in equity: good bot vs bot play in the last computer poker competition dropped down under 20 IIRC, and some of our own tests are sitting in that range. Duplicate seems to give around a 25% reduction in the standard deviation for no-limit. That would give an estimated standard deviation of 0.083 big blinds/hand for average winnings over 80k duplicate hands.

I'd have liked a stronger argument than "It's a tie!"

Edit: And there's some of the actual data. Good!
Suprisingly high values. Either duplicate variance not being computed correctly (I hope not!), or duplicate wasn't giving 25% reduction, or the raw money variance was all the way up around 20 big blinds / hand. Or some combination.

What St. Dev was arrived at? Calling it a statistical tie is not correct though is it? More correct to say, as you did, that the win is not conclusive at the 95% confidence interval.

If the devs could - would you please put the histogram up of results at 100 hand intervals?

around 10bb/100 is a wide lead. telling it is a "tie" is forging the results. (I mean: giving false information or giving a valid information that misleading the audience)

we are all playing poker and a lot of us is practiced deep stack games (100bb+) and we know how much winrate can be expected from 80k hands (I am not mentioning that hands was mirrored and eq chopped). Against a good player it is a wide win (against a weak player 10bb/100 is not a great result ofc). You can tell to the audience you AI played better than a bad player, but if you tell Claudico made a tie against human team we will know it is against the truth and the interest of all poker players (so this forging probably will be well known publicly).

Or even better, the bb result of each hand.

No. That would be great lower bound on exploitability. Anybody who did some poker programming knows that and they also know it's quite a useful bound which hard to get to low values.

Well it was clear for some time he doesn't know what he is talking about anyway. It is just ridiculous but unlike claims about programming it's easily verifiable as ridiculous even by non-programmers.

This is tilting because Holdem is simple enough game that if you throw resources at it you will get a decent AI even if you are absolutely clueless about poker and poker programming. It's tilting the hell out of me even more so seeing how Sam is apparenlty very reasonable guy who is trying to answer questions and gives honest info/assessments while the boss in a project is a major bs'er.

At this point I wouldn't trust the prof to calculate variance correctly. I think the players have their HHs from the match so maybe they could ask someone to write a simple script and verify it (I understand they don't want to disclose them).

I'm surprised that the players agreed to this wording? Maybe one of them can chime in on their reasoning? "We didn't gaf" will suffice

When I saw it was Polk and Donger, I was confident in the humans.

When I saw Bjorn had been added, I grew even more confident.

When I saw Jason Les had been added, I knew humans were in trouble.