This paper shows that imperfect-recall abstractions improve performance over perfect-recall ones, http://webdocs.cs.ualberta.ca/~bowli...bstraction.pdf. The analysis uses both exploitability and head-to-head performance metrics. Those results were for limit Texas Hold'em, but based on personal communication with the authors similar improvements have been observed in no-limit also with respect to the head-to-head performance metric. Since computing exploitability in no-limit is currently intractable, there are no results for that metric in that domain yet.

I definitely think that computing exploitability is important. Many of my papers use exploitability calculations whenever applicable/feasible, e.g., http://www.cs.cmu.edu/~sganzfri/Translation_IJCAI13.pdf, http://www.cs.cmu.edu/~sganzfri/Puri...on_AAMAS12.pdf.

The first approach for computing exploitability in LHE was just developed a few years ago, and involved sophisticated techniques to be feasible, http://webdocs.cs.ualberta.ca/~bowli...ijcai-rgbr.pdf. There are no known approaches for doing this in NL. I suppose the techniques from that paper could be applied for computing exploitability in NL if restricted to one betting size everywhere. Researchers from University of Alberta presented preliminary results of doing this at the AAAI Computer Poker and Imperfect Information Workshop back in 2012 or 2013 on their prior year's NL competition strategies. The exploitability that was reported was extremely high. They did not end up publishing this result, and when I followed up they were still looking into verifying the results to ensure correctness.

It seems like this approach would be totally infeasible for computing exploitability with more than one bet size.

Some posters have talked about computing estimates of exploitability, e.g., fixing preflop strategies and computing "postflop exploitability," computing exploitability within some abstraction, etc. I'm just not sure what the merit of such a value would be. If I say "I computed approximate exploitability using abstraction A and assumptions X, Y, and Z and it was 27.2 BB/100," there are probably two people in the world who would care.

Like I mentioned, I'm also not really sure how feasible these computations would be for strategies based on imperfect-recall abstractions.

It's possible that there is something interesting here, but I just haven't had much time to look into this problem very carefully, and as described above my instinct is that it would not be very valuable scientifically at this point (and would possibly require a lot of time and effort).

What I do think would be valuable would be an efficient general algorithm for computing best responses in games with imperfect recall, which would not be poker specific and could theoretically have broader applications. If I'm able to come up with an algorithm for that, then I would be interested in applying it to NLHE.

''That sounds like a lot, but over 80,000 hands and $170 million of virtual money being bet, three-quarters of a million bucks is pretty much a rounding error, the experimenters said, and can't be considered a statistically significant victory.

But a tie is significant in and of itself: It suggests that Claudico is at least a match for human players, at least one-on-one and in this particular game.''

from the NBCnews article. wow...

I did not make any comments regarding what confidence interval to use, whether it was a "tie," etc. If other people from CMU or the media want to claim it was a "statistical tie" that is their prerogative.

Absolutely hilarious how this was spun as a tie by the CMU team... Oh well, I guess it doesn't truly matter in the long run as computers will inevitably catch up.

This is just pathetic...

Inb4 the professor says "I'm 95% confident that every hand is fifty-fifty".

Addressed your questions in above post.

I stopped posting in the prior thread because I was being harassed about the identity of the developers of a program we played against. It looks like you tacked on a big post at the end there.

Approaches from research on poker have already been successfully applied to other applications, e.g., robust policies for diabetes management, http://webdocs.cs.ualberta.ca/~bowli...2nips-kofn.pdf.

@Sam: Apologies, it was noambrown.

One other point I'd like to add in addition to my above posts is that sometimes it's very hard to prove theoretical guarantees for approaches for hard problems. E.g., there are no bounds for any of the abstraction algorithms for bucketing hands together. It's pretty much impossible to prove anything theoretically when mapping 10^160 states down to 10^14. As far as I know the only "lossy" abstraction algorithm with a provable bound is from colleagues at CMU, and scales only to a poker game with a 5-card deck, http://www.cs.cmu.edu/~ckroer/papers...4-extended.pdf.

It's valuable to prove theoretical bounds on approaches, but it's also valuable to come up with agents for hard problems, like NLHE, which might involve some heuristics/approximations without provable guarantees/bounds.

It's pretty common to present theoretical analysis just for smaller domains, and then empirical results for larger "real-world" ones. E.g., a lot of poker papers present exploitability calculations on small variants like Leduc Hold'em as a proof of concept, but head-to-head empirical results for full NLHE, e.g., http://www.cs.cmu.edu/~sganzfri/Translation_IJCAI13.pdf, http://www.cs.cmu.edu/~kwaugh/publications/nips09.pdf.

Hopefully the theoretical bounds and the scalable heuristic approaches will eventually meet somewhere, but this is not always possible.

I understand where the CMU public relations team is coming from when they call this a "statistical tie". However, I feel like the degree to which that (and additionally the "win vs money wagered" comparison used) has been pushed in the press release that was distributed to media outlets went too far and is being used to color this more favorably for their self-interest at the expense of the human's accomplishment here.

I also want to say that coming down the stretch in this challenge my personal results vs Claudico were often held as a trophy to the media as if it were evidence that Claudico was beating an individual, despite the format and spirit of this competition being a "team". I don't recall the term "statistical tie" ever being used then.

Also I recorded this for 2p2 because I want to share what I thought was a great talk by Doug at the closing press conference of the challenge. He outlined everything quite well and I agree with everything he had to say.

https://www.youtube.com/watch?v=L10zaLtUabY

A University should be held to a higher standard than a beer commercial. Claiming a tie is straight up academic dishonesty/fraud and they should be sanctioned if not fired.

If Barcelona beats Bayern at 4:3 in football that would be a tie too. Because it is not statistically significant.

Moreso when Klitchko K.o-ed Pulev in boxing was statistically insignificant. That should be a tie too.

I suggest to poker media to do not spread the lie about "tie". Only tell the truth:

Facts:
- 80k mirrored and eq chopped hands has been played,
- 7320 blinds won by Brains team,
- more than 9bb/100 winrate against Claudico
- CMU team still consider a p=92% win, and a 9bb/100 loosing as a "tie" which is a joke for any poker player.

(Let me be demagogue a little: any of us would play "This Was A Tie" (sic. professor said this) game life long at high stakes. For example this game means at 200 hands/hour at nl2k, the human player can make 360 usd/hour pre rake hourly rate, which is a nice salary and Claudico would make -3M USD per year against best human players at two table yearly. pre rake this too ofc... It is stupid calculation, but telling lies to poker players about simple results is stupid thing too.)

Would you please post the math for the 95% confidence interval?

Im disgusted.

yeah pretty ridiculous by these profs to consider this a tie.

Seems like the organizers should have said that the humans won, but that there was an 8% chance that the humans had no actual edge, due to the role of chance. That would have made a lot more sense.

When all the hands are broadcasted with holecards up, it should go without saying that the players will not bring their top default game. While 9bb/100 is a crushing winrate, I doubt that this was the full potential of the human team.

Would love to hear one of the players confirm this

I for one am just glad to see that AI developers can be just as delusional about variance, edges, and, "I'm just running bad" as casual players.

Bots are the new fish, imo

Based on the results, what is the % chance that the players are better than the bot?

You think that the players are actively playing less than optimally?

Pretty gross they did this to you, I noticed it throughout the contest myself. Not fair at all especially given what they said prior to hands being played, and after hands being played. You handled it well though Jason, and you handled their bot too

If this result doesn't qualify as statistically significant, how likely was this challenge to generate a statistically significant result?