More bs from CM:
https://www.cs.cmu.edu/news/upping-a...intelligence-0
For one thing, I don't know how I feel about the statement that Poker poses a more difficult challenge than Go -- surely incomplete information makes things more difficult, but there are many factors that add to the complexity of a game and how easy it is to solve.
But that's besides the point -- which is that Prof. Sandholm sells it like letting a hunl poker bot play vs human players is the only/best way to evaluate its strength. In 1vs1 poker, there is a scientific measure that is precise and comparable, exploitability. That's a direct property of a Nash equilibirum for 2 player zero-sum games. And while it may be sometimes tough to calculate it based on how the bot is working/how the strategy is saved, there surely is a way to compute a lower bound of exploitability, just as the two authors in the paper linked above did. Note how these authors aimed at comparing the strength of various hunl bots and used exploitability as the natural measure -- how would you feel if they instead played every bot for 30k hands each and published their results?
How will CMU compare Claudico and Libratus strategies for instance? By comparing the results of both brains vs ai competitions? The article said that CMU is running the supercomputer for around 15 million core hours (compared to 2-3 million for Claudico) -- are they never checking whether running it longer is actually making progress in the right direction when using that many resources? If yes, how if not by comparing exploitability? Is there any theoretical proof that their algorithms converge to equilibirum?
Bottom line is, whenever someone presents you a poker strategy that is supposed to be good/aiming for optimal, and they dont't present you a (lower bound of) exploitability number, then that means they either don't know what they are doing since they aren't measuring what they are doing, or they know that whatever they have done isn't working too well and consciously decide to not to let people know but rather sell it through another way.
Last edited by samooth; 01-05-2017 at 10:05 AM.