Quote:
Originally Posted by Sam Ganzfried
It's not clear to me if it's doable, and what the scientific value is of computing a bound on our exploitability like this. When I have some actual time (i.e., after I have my diploma in hand) I'll think about it some more and get back to you.
Come on, man.
We had this thread in late September last year dedicated to Tartanium7:
http://forumserver.twoplustwo.com/29...25/index5.html. In this thread, several posters pointed out that -- since the bot is playing a static strategy that resembles your best approximation of a NE in 200bb deep hunl -- an exploitability number or a lower bound of expl. is not only useful but imperative to measure how well the bot/strategy actually is. If you go to the last page of this thread, you see that it died with this exact discussion going on and was left unaddressed, eventually.
After a 6 month posting hiatus, you come back and post
Quote:
Originally Posted by Sam Ganzfried
Our 2014 agent Tartanian7 beat everyone with statistical significance in the computer poker competition, including teams that had programmers with expert poker knowledge. Claudico is way better than Tartanian7.
Together with you questioning the "scientific value" of exploitability, this post indicates that for some reason, you and your team are resisting to understand the importance of the issue. punter11235 has made a lot of good posts explaining exploitability and its importance -- it's the most natural test, and it lets you/us actually quantify how well the approximation is rather than relying on relative benchmarks (other bots/humans) with results also being affected by variance. The fact that you beat other bots and that you (probably) lost with Claudico at a 9ish bb/100 rate really tells us nothing other than the obvious (x is better than y over z hands).
I'm not coming here to bring any hate, I have congratulated you on Tartanian7 and I really enjoyed following this challenge as well. It is simply inconceivable for me that you and your team are apparently resisting to understand the entire concept. It becomes even more inconceivable after reading
Quote:
Originally Posted by Sam Ganzfried
There are some limitations in academia that wouldn't apply to people outside of academia. We can only work on approaches that can be published in strong conferences. Often reviewers know close to nothing about poker, and a paper that is considered "poker specific" will just be auto rejected. (Earlier versions of) my endgame solving paper that was linked above for example was rejected four times, before it was finally accepted with scores right around the minimum threshold for acceptance. People outside of academia are free to work on poker-specific approaches without regard to whether they would lead to publication.
How are you going to publish papers based on these poker bots without giving out that number? If the answer is "the referees don't demand it", then it is in line with my own experience in academia (although from an unrelated field). meh
Also, this question still stands:
Quote:
Originally Posted by punter11235
Is there any scientific value in approximating something you don't have a measure for?
In contrast however, you have quoted the Cepheus bot and its exploitability number, implying you understand its usefulness -- which brings me to my last point:
Quote:
Originally Posted by Sam Ganzfried
A big goal of this research is to develop techniques that will apply to domains outside of poker, e.g., national security and medicine, where problems are modeled as imperfect-information games. So we prefer approaches that are more broadly applicable to ones that are purely poker specific.
You absolutely cannot recommend any technique to other domains like national security or medicine without measuring how well it actually works, so as long as you are not giving out that number...
Looking forward to your reply and all infos on how Claudico works.