Not sure if this is the correct part of the forum to post this

But I really need to decide between the two.

These are the main two softwares that managed to "solve" no limit holdem postflop GTO play using cutting edge algorithm like fictitious play.

But which one is more suited for me?

Lets compare to two...

PioSOLVER
http://piosolver.myshopify.com/

Pros
- very fast GTO solver for Holdem:
handles postflop spots with arbitrary starting ranges, stack sizes, bet sizes as well as desired accuracy, give you the GTO frequencies, gives you the range of that frequencies (e.g. bet x% of this combo), solve for all turn and river run outs
- unlimited uses
- can import crEV trees
- one time payment

Cons
- might need a fast computer
- gives you the answer but no interpretation other then pure numbers
- looks kinda primitive (e.g. no visual tree, but has the chart tho), colour scheme abit weird
- no active communities and discussion/sharing of result as of now

piosolver current has basic/pro/edge version for 250/475/1100 USD not much different between them yet in function (will change in future)


GTOrb
http://gtorangebuilder.com/#home

Pros
- much more interactive, flop solution library with a database of solution by the software
- existent community and discussion on the result - extrapolation to theory
- solve for equilibrium like pioSOLVER
- head of site is Alex Sutherland who would makes strategy packs and training videos (however that can be bought seperately)

Cons
- Can't actually be use to solve for equilibrium yourself, you have to submit to GTOrb, it could be chosen to be solved and posted on flop solution library
- recurring payment: 6 Month License - $299/ One Year License - $479


The question is which one would be better for studying GTO frequencies, a product that can allow you to solve everything without restriction, or one that has more restriction in what you can solve, but has a community that can help you get more out of it.

Of course both parties will change and adapt to the market and there could be changes in the usage (I'll keep people updated), but which one would you prefer, taking everything into account from price to quality.

how is this poker theory?

Yea sorry mate, I dunno which forum this thread belongs to...

You could also consider Solving Poker With IPython And Fictitious Play by Will Tipton. I'm not a programmer and was able to get everything working.

Both of these other tools look great but the information premium has to decay a bit for me to be able to justify the cost.

they're both the same, one is a little faster, the other is a little better presented

Was expecting more responses, thought everyone would go crazy over GTO now that its all out there.

How big of a deal is piosolver having flop solver compared to gtorb which seems to have a much more intuitive interface?

Yea i dunno, problem with gtorb is you cant solve flops urself, you have to wait for results...

So as this thread keeps coming back and no one is willing to answer I will. I am PioSOLVER author so take that into account.

We are better for:
-solving flops in general and solving a lot of flops fast in particular
-experimenting with different ranges/bet sizes/stack sizes and seeing how they influence the solution (cause for that you need fast solving)
-pulling general stats about the solutions (like comparing frequencies on various turn/river cards of given lines)
-analyzing the ranges at any point in the tree (cause range explorer with categories etc.)

They are better for:
-For sure for coaching material, videos, commented available solutions, probably still for community (we have more high stakes/nosebleeds users but you won't get them to share anything)
-as to the rest I will let GTORB people to elaborate as I am sure they have more views about their advantages

As to the interface: drawing the whole tree is in my view a mistake and it makes presentation more problematic once there is a lot of information (as most of the screen is taken by empty space between nodes). It is more intuitive at first that's for sure though. In PioSOLVER there is a learning curve but once you get it going you can access more information quicker and export it in useful format as well (for example to CSV to use in Excel/Open Offfice).

This is of course matter of preference. I have users who tell me it's way better than GTORB and I have people who tell me they want the whole tree view no matter what and that's a major disadvantage for us.

So now the question is what do you want. If you want teaching material selected by competent people (as I believe GTORB staff is) nicely commented etc. you would probably benefit from it more right now.
If you want to experiment yourself, see how things change, solve a lot of flops and try to understand how equilibrium looks like then you can't find anything which remotely compares with what we offer.

We spend most of our time programming and it's true we don't have much teaching material nor marketing yet. Those things will happen though over time. It's just our priorities is to deliver the ultimate poker tool first, teach people how to use it later.

I hope that helps

Thank you punter, hopefully the gtorb offers similar input from his point of view. The softwares seem to be cheap for what they do but still a big investment for people not playing midstakes+, so knowing fully where each software shines and has its weaknesses would be very beneficial.

Hi,

I'm the creator of GTORB and I'll try and briefly highlight the advantages of GTORB and I should also mention that there is a third alternative out there called Simple Postflop that I believe has the highest accuracy / fastest flop calculations of any of the three programs in many situations at the moment, although I haven't personally done a performance test recently.

GTORB will be supporting the option to upload and browse/share Simple Postflop solution files in the GTORangeBuilder solution browser UI in our next major release, so I think for people who want to run their own flop calculations while still being able to browse these solutions in GTORB's UI this will be a very strong option that will hopefully offer the best of both worlds (compute solutions in Simple Postflop and browse/share them in GTORB viewer). In addition to having very fast desktop solutions Simple Postflop also has fast and affordable server calculated solutions so you can solve scenarios that would require too much memory to run on your desktop using their servers.

Regarding GTORB's advantages / disadvantages it is fairly different from both Simple Postflop and Piosolver, I'll try and highlight its advantages below. I think in general Punter's comparison was very fair and correct (I have a lot of respect for his work and I have nothing negative to say about PioSolver) so I'll just add a few additional thoughts on GTORBs value.

The flop solution library

GTORB is a bit different from the other equilibrium solvers in that we offer a searchable online library of existing flop solutions calculated by us / other users, all of which are instantly available for browsing (there are a few free publicly available solutions in the library that you can check out without an account if you are interested). We have ~600-700 flop solutions in this library currently with more being added every day and you can browse all of them for $20 which means that for a very low price you can study a huge amount of GTO solutions. This lets you see exactly what starting ranges / game trees other players / pros are using and learn from scenarios that were designed by others. As part of this you can see which solutions are highly rated by other users and each solution has its own dedicated comment thread where you can discuss the results.

The amount of time required to think about and construct a good scenario that accurately models a situation of interest (even if actually running the computation to solve it were instant) is significant so for many users having hundreds of existing scenarios at your finger tips to search and browse can make your study time a lot more efficient by allowing you to leverage the work of others to accelerate your learning. This would be true even if you were going to user Simple Postflop or Piosolver to run a bunch of flop calculations on your desktop.

Additionally, anyone with a GTORB Pro license can submit new scenarios to the library and we solve them and add them to the library if they meet our quality standards. We also make it very easy to do things like take another users solution and tweek a range or the size of a specific bet so that people can riff off of each others work and learn and we've had some really cool examples of this kind of interaction leading to interesting discoveries.

Strategy Packs

We put a lot of emphasis on teaching the concepts of game theory and helping people learn how to efficiently leverage computational GTO solutions to directly improve your game play rather than just on providing the tools to compute solutions.

We offer a bunch of strategy packs that offer a specific set of pre-computed solutions that are constructed by an expert in a particular topic / game type that come with accompanying video that are designed to help people bridge the gap between computing a solution and actually applying that information to improve their game play at the tables. The strategy packs are independent of GTORB and don't require a GTORB license.

We also offer some free videos on our youtube channel and I have a series of card runners video for people who are looking to deeply understand game theory and the nuances of understanding and applying the results of computational GTO solutions.


UI / Accessability / Sharability

All of our content is available in the web in a polished, easy to use, and professional UI. We have a powerful / simple GUI for locking in set strategies at specific nodes, graphically building and customizing game trees and of course browsing solutions. You don't need to download and install anything to use GTORB and you can browse solutions on any computer with a internet connection. You can share solutions with others just by sending them a link. Because its all web based you can also easily have many different solutions open in different browser tabs and take full advantage of all of the features of your web browser.

Hope this helps,

-swc

Well so when we are at the topic of SP it's unfair to make any claims about performance as they still don't show you the whole tree. That means you need to spend several seconds (or more depending how big the tree is) on every turn node you visit and then on every river. That makes performance claims not very reliable because:

-you can't gather information from different branches of the tree without spending several seconds on every turn and and then river; that make questions like "what is average bet frequency in bet/bet/bet line" or "what is the best turn cards ti 2bd barrel" not answerable;
-it also make browsing a pain because there is a pause every time you go to a turn or river
-it makes the results for multi street games not verifiable
-if you are taking additional time for every turn/river you could easily claim 10x performance but again it's not the way you measure things because you can't access the results
-evs calculated that way are not verifiable (because recalculating streets means the strategies are different than the ones you had once you produced ev/strategy results)

That makes the tool very limited because what it does is to give you flop approximation and then it calculates turns/rivers for you assuming that approximation was correct.
This is easy to fix if you in fact solve the whole tree so let's wait until it's fixed and the whole tree is shown before claiming anything.
While I don't have reasons to doubt it works correctly internally and it's just implementation mistake to forget the tree once it's solved I was waiting for fully functioning version before making any comparisons. I've seen a lot of wild claims though which really don't do any favors to potential customers and I am getting a bit annoyed by what I consider misinformation on the topic (and other topics, saves size being my favorite).
Let's wait until the most fundamental functionality of the solver which is actually showing the results in their full form is provided before there is any discussion about performance or accuracy.
As to other features I am happy to answer any questions about specifics and I think we are way ahead on most (maybe all) fronts.

As to the rest of your post I consider it a fair advertisement of GTORB

I am happy to engage in more detailed discussion about usefulness of of particular features or in feature comparisons but maybe it's not the best place for that at least until specific questions are asked by other posters.

I think posts like that are usually the best left for customers/reviewers, on the other hand there is so little information right now that maybe even biased perspective from the authors is better than nothing.
I encourage everyone to ask questions in software specific topics because if the more heated discussion starts here between the authors it will be hard to stop as everyone likes to have the last word and we all believe in what we are doing

EDIT: btw I think the best place for such threads is in software main forum as this one seems to be dedicated to threads about specific products not discussion about them

I think that's a fair analysis of the two (and third).

No, it's not fair. It's misinformation. As I have more and more people coming back to me saying they were mislead about claims made by SP I am going to set it all straight soon.
For now though let just focus on one thing:

You need to wait for every turn in SP. If I were to force users to wait even 2 seconds for every turn I could claim 3x better performance (just tested it, it's going from 0.85% to 0.25% if you give it two seconds on every turn without touching the flop).
This is still very charitable because:

a)you need to wait more than 2 seconds (more like 5-10)
b)you also need to wait for rivers

Another thing is that you can't claim anything about accuracy without seeing the whole tree. I mean, how can you test anything if you can't see (let alone export) the strategy without recalculating nodes? It's not the tree the claim was made about anymore. You can't verify EVs, you can't check correctness of anything but rivers.
If we were to implement waiting on every turn/river we could claim 0.2% exploitablity instead of 1% and that would be accurate (as my experiment just shows) but we don't.

It's either purposely dishonest or just very optimistic overreaching advertisement. Whichever it is it's directly hurting both customers and my business. There are more claims you can find on the internet made by people associated with SP which has nothing to do with reality (like for example claim that PioSOLVER saves are big compared to SP made 3 weeks after we released an option to choose save size while conveniently omitting the fact saving a tree in SP means losing all the turn/river information which people need for analysis/exporting results).

Again I am going to publish benchmarks and feature comparison once I have it but it's too much to leave it here without a comment as in last 2 weeks I had several people coming to me and telling me straight up they were mislead and the software in current form is barely usable for serious analysis.

I wasn't bringing it up myself (nor here nor in other threads) because I wanted to wait for the most urgent things to be fixed to make it possible to compare the two.
Now I see though that not only they are not fixed but claims are made about our product which are either straight-up incorrect or impossible to verify.

Hi, I'm the creator of Simple Postflop and I'd like to correct some of your assertions regarding the quality of our software.

In the past we have ignored some of your claims about our software because we have felt that it is inappropriate for us to reply, but we cannot allow this thread to misinform our potential customers as to the power and accuracy of our software as this unfairly hurts our business.

1. First I would like to be completely clear. Simple Postflop solves the entire game tree including all turns and rivers and store the full game tree in RAM. No shortcuts are taken in our solution algorithm and our nash distance is well defined and accurate as reported. Our nash distance measure is computed based on the entire game tree which is available in memory. We choose not to store the entire game tree to the hard drive to minimize solution size and to reduce the storage costs of our cloud system. Furthermore, we recompute the turn and river solutions for higher accuracy on those specific sub branches of the game tree. This approach is used by top competitors in AI competitions and is completely standard. Given that PioSolver recently copied our approach and gave its users the option to only save the flop portion of the game tree to disk we believe you recognize the value of this approach. We do not implement this by taking any shortcuts and still solve the entire game tree which is the only way to guarantee an accurate nash distance, I hope that your implementation of the feature is the same. It is not possible to compute a nash distance metric without having the full game tree solved.
   
   I know that our speed and accuracy might be surprising, but it is because our algorithm is very fast, not because we are not computing a full solution. We are happy to add the ability for users to save the entire game tree (or flops and turns) but that feature has not been requested by many of our users.
   
   
2. We believe that a number of your marketing claims relating your program to your competitors are clearly false. You have posted that "we are already way ahead of the rest of the market in performance" when it is clear that if you had conducted any bench marks against Simple Postflop that you would find that that was not the case. Furthermore, on the front page of your site you claim that prior to the release of PioSolver it was impossible to answer questions like, "What hands are the best semi-bluffs on the turn? What hands are the best bluff catchers on the river?". Clearly GTORangeBuilder was out many months before PioSolver and could answer these questions. Your website's claims that PioSolver was the first program of its kind, "the first in a new generation of tools moving poker from a game based mainly on intuition to a game based on analysis and math". We believe that any external party who fairly evaluated your claims would recognize that they are simply untrue: existing programs could answer those questions prior to PioSolver's launch, your software was not the first of its kind, and your current performance and accuracy is not ahead with Simple Postflop's.

If a neutral third party would like to do a bench mark comparison using Simple Postflop we would be happy to provide the ability to save a full solution to the hard drive, this has not been our top priority because our users do not often request it, but of course it is possible for us to add.

I believe you. Yet there is some chance you made a mistake which is now impossible to verify.

You don't need to store the tree, it would be enough if it stays in RAM to at least attempt verification.

What is used in AI competitions doesn't matter because we are both way ahead of them.
This approach is ok if you recalculate turns/rivers for enough time. It's however still the case that:

1)you can't verify it because the tree the results are shown for is no longer there
2)if you are allowed to take additional several seconds on the turn (and some miliseconds on the river) you could easily claim 3x lower exploitability and be correct (that's the tree user would see);

I think I've learnt about this idea from this guy in this post:

http://forumserver.twoplustwo.com/sh...3&postcount=88

Yes, for storing but then there is clear explicit cost that you would have to recalculate. We even given approximates how long that recalculation could be (about 200-300ms for a river, several seconds for the turn).

I believe you.
Still:
1)it's not verifiable until you show the whole tree
2)Doing it this way is the same from user standpoint as not solving turns/rivers to carefully and then "catch up" when recalculating. Do I think you are doing this? No. It doesn't matter though becuse form user experience perspective it's the same thing: you still need to spend X seconds*amount of branches you want to visit.

I claimed that in response to wild claims which were coming from your camp.
I also did benchmark it and the claimed performance was about the same (granted it was HU case with huge ranges so it's possible/likely it would be different for small ranges).
To be able to compare user experience though I made a "recalc every turn" feature and observed that I can claim way better exploitability (3x or more for some cases I tested).
In absence of ways to verify if you are calculating the whole tree correctly we can compare how long user's action takes, for example seeing one river on chosen runout.

It didn't solve flops back then, so no it couldn't.

Was there a flop solver available back then?

We can't know that. We can measure how fast you can gather info from the tree, in that we are way ahead and that is what matters for most usecases (all of them which go beyond the flop). I believe you can fix it of course and I was waiting for you to do so to run benchmarks but then I've heard claims about performance and things like saves etc. which you can find if you google around.

I am more than happy to let neutral party to benchmark and verify stuff. As it happened in chess it will probably happen in poker that we will have those reviewers who do regular updates and rigorous measurements. That's not the point of my post though.
I also don't doubt that you are a good programmer and honest person although I do think you and people associated with you claim things which are not verifiable and misleading.
From user standpoint it doesn't matter what you do internally. It matters how fast you can see the results at given accuracy. That's the one thing you can measure and one thing you can use.
I wouldn't bring SP up at all but once I see very reputable guy (Alex Sutherland) claim something about performance it's a problem for me because that's a claim directly about my software which I am unable to defend against because it's also unverifiable.
I then made claims which are verifiable (how long it takes to do typical use case at given accuracy).

So yeah, if you want to claim anything about performance you need to measure something that's measurable.
Tone of my posts are as they are because I listen to what my users tell me. Even the ones who were decideding between the two and went with PioSOLVER told me it's "despite it's supposedly 3x" slower than SP. Then you google and you see similar claims.

As to your comment about copying. I refuted this one but it's really all pointless. Everybody knows all the ideas here: make the fastest solver from preflop and all the cool features. The only question is who delivers it faster/better. I won't claim that you copied this and that from me (like range explorer or w/e else) because the idea is obvious for anyone competent. It's as obvious as the small saves are (although this one is technically way more challenging).

This argument appears ridiculous to me. Who cares about speed? When they are close enough to insignificant compared to the time required to build and analyse a single solution. So you're arguing about some 'marketing buzz' that isn't relevant to real users, only unknowledgeable new users.

The speed and ease to build and analyse solutions is the useful metric not solution speed. Competing over this instead would lead to claims worth making.

So let me elaborate why verification is important and good intentions don't matter when you make claims.
Simple Postflop is claiming that you can round strategies on say flop and turn and instantly have exploitability of those changed strategies, like here:



The problem is that if it would be possible to calculate exploitability of the whole tree after changing strategies in one place without having turns and rivers anymore it would be the biggest breakthrough in poker programming in history which would make solving from preflop possible in about 8GB of RAM. You would just keep the flops in memory, change strats, see how exploitability changed, rinse repeat.

So there are two possibilities here:
1)you made something revolutionary which everybody else in history of poker programming missed which would make solving from preflop possible on your run of the mill laptop

2)you claimed something incorrect

Of course we can't know which one is it until we can see the tree and verify.

Now, my users tell me: "punter11235 SP claims not only better performance and accuracy but they can calculate new exploitability after strategies on flop changes in miliseconds without having turns and rivers in RAM anymore, how comes you can't do that?".

I am ready to admit defeat on this one as my implementation of strategy rounding requires walking the whole tree to calculate new exploitability but you can see as everybody should be very skeptical here.

Almost everybody. That's why the argument about speed is made all the time by both SP and us. It's the single most important feature because it's the difference between being able to make our personal flop library of several hundred flops in 2 days or 7 days. Or spending 2 minutes or 5 minutes per hand when going through spots after your session.

I have a lot of real users who pay me real money and expects real features. Fast solver is number 1 thing which excites most of them.
If it wasn't important nobody would claim anything in that department.

This is respectable view and yeah, I agree it's very important. Keep in mind though that a lot of real people are buying very expensive real computers to calculate more flops faster. Making the solver 2x faster means several thousands dollars savings for them.
Everybody has different views but I am in quite good position to judge it hearing feedback from hundreds of people. Speed is very important thing for many use cases.

What do you think would be the better soft for build preflop exploitive strategy?
CREV? or PiSolver ? GTORB ? Simplepostflop?

Thank you

Again I'm the creator of GTORB so this may be biased.

To accurately compute an exploitative strategy you need to know what your opponent does at every possible decision point in the game tree. For multistreet analysis it is generally not going to be possible/practical to manually enter a strategy for your opponent because there are so many decision nodes (every turn and every river card may lead to different decisions). Because of this, practically speaking to compute a true exploitative strategy you will generally need something that fills in the majority of your opponents strategy for you, while you then pick a few nodes to manually enter a weak strategy that you want to exploit.

You can run a minimally exploitative calculation where for example you force your opponent to c-bet a specific sub-optimal range. The solver than will solve the game tree for an equilibrium under the restriction regarding the c-betting range thus filling in the rest of your opponents strategy. Simple Postflop was the first to do this but I believe GTORB/PIO/SimplePostflop all offer this functionality now. I have a free youtube video on the theory behind this approach here: https://www.youtube.com/watch?v=xfMmaWO42o8, at the time I made that video Simple Postflop was the only option for these calculations so I use that in the video but of course now you can use whichever tool you like (I personally have done it in GTORB and Simple Postflop). This approach is 100% robust to your opponent playing sub-optimally at the decision points that weren't manually set. That is if the minimally exploitative strategy EV is 4bbs, but your opponent actually plays the turn incorrectly (even though you assumed he only c-bet incorrectly) your EV can only go up.

You can also run a maximally exploitative calculation by instead solving the entire game tree for a GTO strategy first, and then after that locking in a sub-optimal c-betting range and then computing a maximally exploitative line against that. This will generally generate a higher EV line than a minimally exploitative strategy if your opponent c-bets sub-optimally and plays GTO at every other decision point, but if your opponent c-bets suboptimally but does not actually play the GTO strategy at other decision points you can actually lose money playing this maximally exploitative strategy. It is not robust to your opponent playing sub-optimally at nodes where his strategy was not manually set. For example the strategy may suggest that you raise c-bets a lot to exploit his c-betting range but if his reaction to a c-bet raise is not GTO you could end up being exploited yourself and losing money. I think that both Pio and CREV offer this approach but I have not attempted it myself in either of them. I know that GTORB does not offer this.

Finally I should say that I tried the CREV equilibrium solver beta very early on and could not get it to work, I have not tried it since, so it is possible that they do offer minimally exploitative solutions now but they did not 2-3 months ago. The older CREV functionality could do maximally exploitative calculations as long as you manually entered a strategy for your opponent at every single decision node, but to actually consider a real strategy where your opponent might play differently on every possible turn card and river card, this would never be practical, you usually had to assume either that specific turn/river cards came, or that the opponents strategy was reasonably similar on different turn / river cards. I would recommend that you download their free trial and check out what exactly their current capabilities are.

Hope this helps,

-SWC

Hello again!

punter11235,
We've run a battery of performance tests to back up our claims that we would like to present to the community. I think the results of the test will help people to asses the honesty of the phrase "we are already way ahead of the rest of the market in performance" that you made prior to your most recent update. I am aware that you released an update today that improved performance, and this test was done without that update.

Tests were done on the same computer(i7) and with the same conditions for both programs - I think that people can trust our results but we are happy to provide a license to our software to an independent third party if anyone wants to verify these results.

The summary is that the EVs of calculations are the same while SPF is 2.9-4.3x faster than PioSolver to the same Nash distance with full hand ranges and 10.8-16.1x faster than PioSolver with tighter ranges. Thus even after your recent update which duplicates some of Simple Postflops optimizations your program will become closer tou our perfomance. Our full data with complete screen shots is at the bottom of this post and speaks for itself.

First I want to answer some of your other statements.

We agree completely

SPF has been out for more than three months and no one has reported any errors in calculations. We have spent a lot of time testing our program results and comparing them to other solvers. Having a full tree allows one to check the nash distance but users don't need to do that, they just need to work with results, analyze them. At the moment I have no reason to doubt in the correctness of calculations. You can note that in our performance comparison above the equilibrium EV results in SPF and in your program match up to 3 decimal places. Practically speaking it would be impossible for this to occur due to a mistake in our solver.

The only possible claim you could make at this point is that you believe we are misrepresenting our nash distance, which I suspect you know that we are not. We are also happy to have a third party independently verify that if required, but at the moment no user has ever expressed this concern, you are the only person who has suggested that our program is illegitimate.

This is our current GUI architecture. It has some benefits and drawbacks but it is important to understand that it doesn't affect the performance and accuracy of the flop calculations strategies.
I believe that here you are tactfully trying to present PioSolver in a favorable light as prior to today's release your solver was 10x+ slower with tight ranges which is why you focus on comparisons across huge ranges.
Again, in fact the performance of your program was 10x times lower in some cases at the time that claim was made and I believe the comparison that was made stating that it was only 3x slower was generous. I see that today you've released an update that includes some optimizations to speed up the calculation but the same optimizations were included in SP two months ago. Instead of understanding why people are dissatisfied with the performance of your program you have attempted to discredit our software. I think it's not very ethical and honest of you.

It's clear that you don't understand how rounding feature works in SPF. We don't calculate flop exploitability in milliseconds. After the calculation of GTO strategy, program just takes strategies and rounds them up to specific %, then solver calculates Nash Distance for the rounded strategy upon the whole tree and sends the results to the client.


Bench Mark Results

1. Full Ranges, 1 sizing, flop.
PIO:
EV Player 1 - 2.711
EV Player 2 - 3.289
Nash Distance - 0.01
Time to reach Nash Distance - 726 sec
SPF:
EV Player 1 - 2.713
EV Player 2 - 3.287
Nash Distance - 0.009
Time to reach Nash Distance - 251 sec
Screenshots:




2. Full Ranges, 1 sizing, flop.
PIO:
EV Player 1 - 2.743
EV Player 2 - 3.257
Nash Distance - 0.002
Time to reach Nash Distance - 1004 sec
SPF:
EV Player 1 - 2.743
EV Player 2 - 3.257
Nash Distance - 0.002
Time to reach Nash Distance - 232 sec
Screenshots:




3. Tight Ranges, 1 sizing, flop.
PIO:
EV Player 1 - 2.597
EV Player 2 - 2.403
Nash Distance - 0.002
Time to reach Nash Distance - 823 sec
SPF:
EV Player 1 - 2.597
EV Player 2 - 2.403
Nash Distance - 0.002
Time to reach Nash Distance - 51 sec
Screenshots:




4. Tight Ranges, 2 sizings, flop.
PIO:
EV Player 1 - 1.763
EV Player 2 - 4.237
Nash Distance - 0.005
Time to reach Nash Distance - 1145 sec
SPF:
EV Player 1 - 1.763
EV Player 2 - 4.237
Nash Distance - 0.005
Time to reach Nash Distance - 106 sec
Screenshots:



5. Tight Ranges, 2 sizings, turn.
PIO:
EV Player 1 - 4.363
EV Player 2 - 7.637
Nash Distance - 0.003
SPF:
EV Player 1 - 4.362
EV Player 2 - 7.637
Nash Distance - 0.006
Screenshots:

Because it's impossible to verify anything more than simple river games as you don't show the whole tree. It's also not true that nobody reported any errors. For example:

http://forumserver.twoplustwo.com/sh...8&postcount=76
http://forumserver.twoplustwo.com/sh...7&postcount=18

First is an error in EV in simplistic river game. 2nd one is you claiming Nash Distance after changing the strategies without having rest of the tree anymore.
I know silly bugs happen all the time in programming but that's why we need ways to verify. It's possible you made a mistake. Maybe there is 2% chance of that happening. Do you want people spending hundred of hours of analysis without nay way to verify that 2% case didn't occur?

Nobody has reasons to doubt nor to believe because it's impossible to verify anything other than simple river games as you can't see what's there.
The nature of the game is that you can claim almost any Nash Distance on say the flop and then recalculate turns/rivers well enough to be correct about the claim. It's not the claim about the tree you have though and it means the EVs are going to be different than displayed if one were to follow the strategy.

I claimed two things:

1)it's impossible to verify the results without seeing rest of the tree for multistreet games
2)Your performance numbers are dishonest because for any usecase you have to wait additional time for turns/rivers; if we were to take that additional time we could easily claim at least 3x better accuracy and still be faster

How can we know?
Also by "flop calculations" you are suggesting it's the whole tree while from the user perspective if we browse node by node there is several seconds waiting period when you enter the turn. If I were to implement that waiting period I could claim way better Nash Distance.
Again I don't believe you are doing it but it doesn't matter. You could do that, it's the same from user perspective because they have to wait for information.
That's the reason I started this discussion. Customers told me they were mislead by your claims about speed because you claim solved trees fast but then you need to wait a lot to see information from anywhere but the flop.

I don't have a Simple Postflop license (I seem to remember that one was promised to me prior your launch but never delivered) so what I did was to run some cases with my users who tried both.
Yes, we tried big ranges (not only 100% though) on two computers and we measured the exploitability. Both cases were run after your speed improvement release and the results in time were close to the same.

I didn't know what kind of optimizations you are doing because there weren't any information/benchmark on your website so I run what was practical/common case then.
Again refer to my main points here:

1)you can't measure performance for flop calcs in SP because it's impossible to verify if claimed accuracy is correct at time of being given
2)performance of anything else than seeing flop strat (like seeing some turns and then rivers on this tree) is overstated by big factor because additional waiting period

Because I know that I can't claim anything which is not verifiable so I made effort to provide results I am making claims about.
There is also more to performance than showing flops alone. If I were to claim flop approximation alone I could easily make it an order of magnitude faster but then nobody in the right mind would believe me (because turns/rivers) would be recalculated.

You realize that if you do that for the whole tree, then forget turns (as you do) then recalculate turns with rounding the results could be different than at the time the claim about exploitability was made?

You are comparing full tree calculation to flop approximation.
This is not apples to apples because we don't wait 5+ seconds on every turn. I made this point several time already and I think it's obvious one. You can only measure things which you can verify. If you want to measure whole tree performance - show the whole tree.
For now it's only possible to measure what really happens in your program which is additional waiting period when browsing. This period is spent calculating stuff. If I were to add this waiting period I could claim better exploitability.
Your performance is based on a tree the user can't see anymore. What good is that even if they believe you are 100% correct if they still need to spend time to see it?


You realize that in this case we are spending most of the time not improving anymore?
Yes, out algorithm is not the best close to 0 and sometimes stalls without additional help but accuracy in both is already very good and improvements from say 0.25% to 0.1% aren't that important.

If you let it run for say 5 minutes or 10 minutes on bigger tree that would be a better comparison. With your methodology you could claim 1000x faster if we stalled at 0.01 and you went to 0.001 (as you could wait as long as you wish). This is extreme example but it shows a bias in the way you are benchmarking it. What tha benchmark shows is that you are way faster at going from almost perfect accuracy to 0 not how fast you are solving the tree to what people consider very good accuracy.

Case in point.

Yup, you are exploiting the fact that we don't go as close to 0 as you do (or at least that the progress is very slow in that zone) again.

So yeah:

1)you are measuring things impossible to verify
2)you are not measuring things which are possible to measure like browsing the tree because that's super convenient for you but super annoying for the users
3)You run all your benchmarks to close to 0 accuracy which we are bad at. I would happily concede that your solver is better at running to 0.001 instead of something practical like 0.25% but someone need to verify correctness of you results first which is of today impossible to do.

I mean this is a common fallacy in statistics. Progress from 0.001 to 0.000000001 is a x1MILLION progress. You are exploiting this here to the end and yeah you've found a weakness of our algorithm which fortunately has 0 significance in practice.
The best one is the turn case which with 1.3 version reached 0.2% in 20 seconds, 0.1% in 33% seconds but then had trouble going to 0.
We expose some settings to make behavior close to 0 better for people who want it but it's true it's not very good.
It still doesn't make your benchmark methodology honest though. You could easily extend it to 100000x faster if you wanted.

Let's find some independent reviewers who can both verify results (once you make it possible) and measure speed on some practical cases.
I've spent a lot of time making my results verifiable, exportable and possible to present in all different ways. Performance suffered a bit but was still very good which would be apparent from your own benchmarks if you showed the whole log instead of the end where the solver is not improving much anymore.

You are also not measuring pauses on the turn/river. This is great practical importance because it means waiting. In your standard 3bet pot for example there are 7 turn exits and 49*7 turns. Adding 3 seconds on every one of them is equivalent of solving for 49*7*3 seconds longer (in accuracy terms not user experience terms). That means we could claim 2x, 3x maybe even 5x accuracy if we were to implement that waiting period.

So yeah I think you are not comparing apples to apples and are exploiting particular weakness of our algorithm (which is behavior close to 0) which I never hid.

You are not measuring things people use the software for and not what they except when they see such claims.

Maybe you could comment on the following statements:

1)The same strategy on the flop may produce the whole tree exploitable for 50% of the pot, 5% of the pot, 0.5% of the pot or 0.05% of the pot depending what's there in the rest of the tree

2)That mean that any claim made about exploitability must be made about the whole tree unless you are saying that "This flop strategy is exploitable at max X with perfect play from there" but this is not what either you or us measure

3)That means your claims about flop performance are not verifiable because you then recalculate turns/rivers

4)If someone wants to browse the tree, compare strategies, see river strategy or anything practical beyond the flop there is additional very significant waiting period

5)If we were to implement that waiting period and usse it for calculation we could claim very significantly faster solutions (often in 3x or more range depending on the tree) and that would be indistinguishable from user experience point of view: the person browsing and looking at the results would look at a solution calculated to the same exploitability in both

6)If someone were to believe your claims from this thread and went with benchmark results you presented they could expect 3x faster performance for 100% vs 100% range while in reality the only thing they could see moving faster is the exploitability number once it's very close to 0; you solving the whole tree underneath has 0 practical importance because it's not possible to see it;

Once you are making comparisons you need to deliver verifiable sensible measure which you are unable to do. Instead you deliver benchmarks based on known statistical fallacy of comparing very small numbers (in absolute sense) and claiming huge relative difference between them. If someone were to go with your claims in this very thread they would expect 3x times faster hand analysis/browsing/collecting data/whatever else you can use the solver for.

That's what I meant when I first claimed misinformation and dishonest claims and I still stand by that.
I also invite you to point at my my statements about SP which aren't accurate because you go a bit overboard with me accusing you of not doing thing correctly. I am saying you are advertising dishonestly and are making misleading claims in relation to my product. Not that you solve anything incorrectly (in fact from what I've seen everything looks correct but it's not possible to make any serious test as it is right now)

For the user that wants to explore ranges on different turns and rivers, I can, as much as I try, not imagine any benefit in having to wait for calculation again for every new card. I might look at a certain board runout, arrive at the river and then want to have a quick look again at how the ranges looked like on the flop ... which, with PioSolver is pretty much just one click, but with SP, means being forced to go through the recalculating process again when I want to go back to examining the river - even when it's the same turn + river runout ! It is irritating (trying to use as neutral a wording here as I can; not here to offend anyone !). I have both PioSolver and SP, but tbh, this one feature of constant re-calculation thing was enough to lead me to me hardly using SP since then and do all my solving work with Pio.

There are other reasons why I personally prefer Pio, but this one is probably the most important one, I see it as a big disadvantage of SP vs Pio. Seeing that you write here that this has "benefits" as well, and I can see none: would you please elaborate what those benefits are, and also, do you perhaps plan on changing SP in such a way that no recalc is needed anymore (as the tree clearly is in RAM, from the space it takes) ?

Also, while I'm at it: I also did a performance comparison, but have to admit it was 2 months ago with the then current versions of both programs, and let them both run to a Nash distance that makes sense in practice, I think it was like 0.3% of the pot or so, with just one bet size (and since I mostly play HU, yes, the ranges were pretty wide for both players). I found Pio faster then, but both programs have changed quite a bit since then, so my results from then are likely not that relevant anymore, admittedly. I just don't think that SP was ever 3- to 10x faster than Pio, when used to a Nash distance that most users probably work with.

Interesting.

Using a solver to analyse hands after a session seems terribly inefficient to me. Also whether it takes 2 or 5 minutes to solve doesn't much matter, it's not like one can't be doing other things while the solver is running. Such use suggests to me, that these users demanding speed aren't using the solvers efficiently.

But perhaps I did underestimate another significant demand for speed, which is possibly a large part of your consumer base. Bot makers & those building solution libraries for availability to make use of in real time.