Hi all,

After reading again the thread and speaking with some players, it is clear that the most recurring criticism concerns the way we handle the term “GTO”. We would like to apologize for misusing the concept of GTO and creating some inaccurate impressions. This was not our intention. Your comments and feedback have made it clear that our statements have been misleading in some cases, and maybe bold in some others. You have made us realize that the approximate or misleading way in which we have referred to GTO is actually a disservice to the strong core value of PokerSnowie.

With this post I would like to be clear and specific about what our statements are in regards to GTO, exploitability and the strengths of our product.


1) Pokersnowie does not play GTO

Going forward we will ensure that we are as clear as possible about the fact that PokerSnowie has not solved GTO but that its training framework leads towards a balanced and un-exploitative play.

Furthermore, we acknowledge the fact, as already mentioned, that we don’t know how far away from GTO we are, and that there is a lot of improvement that can be done in that direction. We will continue training and improving our AI with the goal of ultimately making it converge towards a GTO solution, if this will ever be possible.


2) PokerSnowie has limitations

PokerSnowie definitely has some limitations introduced by our design constraints, like for example, allowing only 3 bet sizes. These limitations clearly contribute from keeping us away from GTO and create some level of exploitability to our AI. A couple of current known examples are the following:

• PokerSnowie does not play well against very unusual bet sizes. It has little experience playing against extremely small bets (0.1 PSB, for example) or unusually large bets. Because of this, it’s possible to find spots where PokerSnowie is clearly folding too much or too little or showing too much or too little aggression.
• In a given spot, PokerSnowie currently only uses one bet size. It chooses what it believes to be the best bet size out of 0.5 PSB, 1 PSB, and 2 PSB rather than a mix of all three. A GTO player would not have this limitation.

3) PokerSnowie does have leaks

Thanks to people’s feedback and internal analysis, we found several leaks in PokerSnowie's strategy. We have fixed some of them by reviewing and changing our learning algorithm. However, as PokerSnowie will never be perfect, this will be an ongoing process.

We also know that the complexity of the game is such that even with the extensive training we have made so far, there are still uncommon situations and spots where the AI is not yet very consistent. As an example we can mention the multi-way pots: they are not only less common, but also significantly more complicated than heads up pots. It’s safe to say PokerSnowie is more likely to suggest a suboptimal line if there are many players in the pot.


4) PokerSnowie is a training and coaching tool

PokerSnowie is not meant to be the ultimate GTO engine, and it will never be. PokerSnowie is meant as a tool for improving certain aspects of your game by, amongst other features, identifying spots where you likely made a sizable mistake. It’s a tool meant for training.

• In most common and realistic situations, PokerSnowie plays very well and most of its recommendations are very strong.
• PokerSnowie can help beginning players see how to construct their ranges and eliminate major leaks, whereas expert players will learn more detailed concepts such as which hands are likely part of mixed strategies and what seemingly weak hands make good flop raises because they retain their equity well.
• PokerSnowie allows players to experiment with new ranges or expensive lines risk free and see how a strong opponent would likely respond.
• When PokerSnowie marks a hand as a “blunder,” there’s a good chance it will be right and you’ve made a significant misplay against a strong opponent. PokerSnowie excels at recognizing situations where one player's range is significantly stronger than the other player's range, and many of these situations aren’t intuitive. Analyzing blunders prevents you from making the same mistake again in a similar spot as well as it may help you to determine less obvious but nevertheless widespread leaks in your game.
• Since we believe PokerSnowie is a strong opponent, you’ll likely learn new concepts simply by playing against it and seeing how it approaches certain spots. Just playing against PokerSnowie will likely help you find leaks in your game you didn’t know you had.

5) AI improvement

The training towards a GTO solution is continuing and will continue for a long time. We don’t know when and if we will ever converge to it, but we do know we are still improving. In particular, very soon we will release a new version of our AI, which solved some leaks and improved some aspects of the game. We are also considering to add more bet sizes and we are working to remove some more design limitations. All these efforts will increase the AI consistency and will improve its un-exploitability. Once we will be ready to release, we will publish an article explaining in more detail what the improvements are.


I hope this post clarifies our position and that you will accept our apology for creating worries or confusions about PokerSnowie’s relation to GTO

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

You are right Barberousse,

maybe not "many", but for sure there are some ways to crush PokerSnowie.

If you use some leaks and limitations of our AI, you would be able to exploit it and win quite a big amount of BB/100. Obviously in this case PokerSnowie would most likely rate you with a very high error rate, therefore what you are reporting makes perfect sense. We fully aknowledge that and as mentioned we are working to remove as many leaks as possible and to improve the consistency of our AI in more uncommon spots.

I just would like to mention that in any case, these strategies that exploit Snowie, are typically strategies you would never use to play online against a real opponent who adapts. Therefore they are a "good exercise" to exploit PokerSnowie, but they are not very useful for the final purpose to use our software as a training tool.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

Hi,

what we mean is that we never add piece of code in the AI in order to decide which move should be done in a specific spot. We use expert knowledge to set up the training environment, to change the learning algorithm and so on.

Since the objective of our AI development is to bring it to an un-exploitative strategy, in theory PokerSnowie should develop a strategy able to defend against any agent.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

Generally, I have to say that I welcome this statement from Roberto and it seems a serious re-think over the loose use of language in regards to what PokerSnowie is and what it isn't...

One warning...your still doing it i.e loose with your language..

I'm not sure you can prove (or at least shown) your method of training your AI even provides a road to a GTO solution. The core training idea may sound reasonable in principle, certainly as a method to improve PokerSnowie's ability, but you could be for instance converging to a local minima. In fact, I would suggest the limited different types of bet sizing is certainly important deficiency that will ensure you will never converge to it. Even with more bet sizes, doesn't ensure you will be on the right track.

It also seems like there is still debate in terms in regards to multi-way games, especially ones where collusion and collaboration can occur. By this I don't just mean PokerSnowie and its method of learning, but more generally.

I would junk the whole GTO spiel and focus on proving / selling the AI as tough, difficult to exploit, etc.

Great, these changes and apologies are highly appreciated.

I still don't think much of snowie's AI, but I'm sure we'll all be more receptive toward its potential when your blog and marketing strategy will stop looking like a scam operation.

I also agree with what oracle said.

bye bye poker

no more money in poker, everybody uses Pokersnowie

Hi omgPirate,

PokerSnowie does not adapt, it plays a fixed strategy, independently from the playing style of the opponent.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

Give a straight answer to the annual computer poker competition challenge please. This would be a pragmatic way to show that the AI is relatively strong.

Interesting article, thanks! Actually I didn´t found anything new / surprising, and I (still) think Snowie is a great and interesting product. But sth. else made me wondering:

"Thanks to people’s feedback and internal analysis, we found several leaks in PokerSnowie's strategy. We have fixed some of them by reviewing and changing our learning algorithm."

Can you explain that a bit more (w/o revealing any "secrets")? For obvious reasons it is / was not possible to fix the "strategy", now you said you fixed the "learning algorithm". How can that fix improve Snowies strategy in short time - when it´s current strategy is the result of billions and billions of played hands so far?

So is all the back and forth basically just over whether or not the advertising on snowie's site is factual, or is it over whether or not snowie provides anything of value at all?

It seems like the brute force continual adjustment until you reach a solution that can't be beaten is the way that I've seen some gto stuff reached in books. Like Tipton's, right?

Snowie only uses 3 bet sizes, so its got major major leaks. But does it approximate gto in a nlhe game that only uses these 3 betsizes? There is value in this solution if it does. And in the method they used to get there.

Of course snowie can't play other bots when it has these huge leaks. Asking it to compete in a competition vs bots that will own it is silly. But there is still plenty to learn from snowie if its method of finding an approx solution to this version of the game is sound. And it likely shows just how daunting and far off the true solution is.

Hi David and nomoD,

Snowie uses only one bet size per situation and can only choose between 3 bet sizes. This is a clear limitation by design. As we stated, in our current improvement plans we are adding more bet sizes.

On the other hand, during training Snowie needs to defend against various other bet sizes, including very small bets and large overbets. As a result, Snowie has learned to defend against a 70% pot bet, for example, and is not surprised by this amount.

Also, for bet sizes that Snowie has never seen in training (like a 1% pot bet or a 100 times pot bet), algorithms have been implemented to handle these situations as good as possible.

Please notice, that Snowie actually sees the exact bet sizes and makes a difference if the bet size is 60% or 61% of the pot.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

Please stop beating around the bush and address the important issues:

1) Submit a Snowie agent to the annual computer bot competition which is open for registration now.

2) Measure the exploitability of Snowie.

The fact that an expert on computer poker posted here several times politely explaining how it is possible to measure the exploitability of an AI and you ignored it, already shows how dodgy you are.

Also you keep avoiding my questions about the annual computer bot competition.

For those that want to see it, Snowie looks like a scam operation no matter how much you reword your marketing on your site or here.

Hi Everyone:

I want to change the subject a little and get some feedback from those of you who have been participating in this thread. And here are three questions:

1. Concerning the technical issues raised with PokerSnowie, are these being addressed well by the PokerSnowie team and does everyone agree that good progress has been made?

2. Are the claims now being made relative to the PokerSnowie product realistic, or is there still stuff on their website which may need some rewording?

3. And most important, do we agree that PokerSnowie is in fact a good product?

Thanks and Best wishes,

Mason

Mason,

With all due respect:

1. What "technical" issues? As a piece of software I am not really aware of any huge problems with snowie. Not sure what you mean.

2. No, not really.
a) On front page of their site it stills says "one of the best GTO approximations on the market"
b) "which will help you play an unexploitable strategy"
c) "Train with a very strong GTO player with "Challenge PokerSnowie""
d) "...developed PokerSnowie into the Best NL hold'em player in the world."
e) "Get the free PreflopAdvisor mobile application for a sneak peek at PokerSnowie's world-class game theory optimal approach."
This is JUST on the front splash page you get when you go to pokersnowie.com

I'd rate their cleanup effort as a D+

3. This forum (Poker Theory) is in my opinion, not a forum that should be hawking or promoting a piece of software. 2+2 has forums dedicated to promoting things and advertising, as I understand it, and it also I believe has software forums that have threads dedicated to individual software companies and products.

This particular forum is for the discussion of poker theory and typically we prevent people from coming in here to discuss their commercial software.

I agree 100% with rustybrooks

I'd be interested in #1. The problem with #2 is that it won't tell you anything you didn't already know (namely, that even the best nlhe AIs today are absurdly exploitable)

*EDIT* And I'm not saying PokerSnowie fits in with "the best" *EDIT*

Yea, but regardless, it's funny that the Snowie representative is so shamelessly ignoring this question .

Hi,

just a quick note to inform you that in fact we are making a full review of our website content in order to reflect our recent statements on 2+2. Give us some time and I'll update you next week when we believe we are "good".

About the bot competitions, we are evaluating it and we will get back to you at a later stage.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

1. I'm not sure about which issue we are talking about. If it's about the RNG/cheating issue, then I personally think snowie's measures are rather satisfying though not perfect (we still have no idea whether the RNG is fair or not unless we go through heavy statistical analysis).

However, I don't think this is such an important issue right now. Most people here don't believe that snowie is strong in its current state, thus they don't really care whether it's cheating or not.

2 & 3: What RustyBrooks said.

Hi again,

in fact our marketing team was faster than I expected. In relation to point 2, as we promised, a first bunch of amendments on our website related to our GTO claims has been done. In particular I noticed that also all the RustyBrooks items listed in point 2 were part of this first revision.

We will continue our revision next week and we remain always open to evaluate additional feedback on this matter from any 2+2 user.

About point 3, I agree with RustyBrooks. In fact we are not using this specific forum for advertising purposes and we never requested it. We came here because some people had question about PokerSnowie and because Mason and 2+2 kindly invited us to address them. Though, we will be very happy to answer any additional technical or theoretical question related to PokerSnowie.

Thanks for that. The RNG is the standard C++ random generator. I appreciate the fact that since our software is not open source, and we are not monitored by any third party, you don't have a way to make heavy statistical analysis on it.

On the other hand we do believe that the measures we took and announced, and which will be available in our next 2.6 update, should at least give you the possibility to check that PokerSnowie is not cheating in any way. For the future, we are open to your suggestion about a way to monitor and report the results of the challenges that our users are playing against PokerSnowie.

Best Regards,

Roberto Gobbo
CEO - Snowie Games Ltd.

Just to confirm, if 2+2 was to accept advertising from PokerSnowie, this forum is not the place where it would go.

Best wishes,
Mason

Your question #3 in your post above, asking whether or not it's a "good product" amounts to asking for testimonials, so I'm going to have to disagree with you a bit.

It's much better. I wouldn't quibble too much about the rest.

I honestly don't think most people are particularly worried about this.

On a semi-related note, I'm curious what you mean by "standard c++ random generator"? Do you mean, like, drand48? If so for training purposes you might want to look into a better RNG, I think the standard c++ RNG actually has too short of a repeating interval for use in hard core simulation.

The absolute best way to remove the chance of cheating is to have your software not generate cards at all. Ideally you would have it set up so that it has no more information than any poker "client", i.e. it gets told it's cards and the board, bet sizes, etc, but all dealing and scoring is done outside the client. This way you don't have control of the cards. Provided you use one of the more standard means available to do this, it would also let you participate in matches against other AI agents.
[/QUOTE]