I was wondering about the final goal of this is? is the goal to get this to some sort of neat package (ie: other such programs with a single click shortcut to get started?) or is it always going to be a few different apps with a few different things to do to get it running?

Aside from agreeing with your post I also want to clarify our position on site support - we do want to support as many sites as possible. We are just concentrating on other things at the moment and for the foreseeable future, but would be happy to assist others with writing a converter for their favourite site. Of course, existing converters could easily be integrated.

Yes and yes. The final aim is to have multiple (meaning at least two) programs that are completely independent. fpdb-main could then be used with another HUD. Equally, fpdb-hud could be used with another tracker (such as PT3). Modularity makes things (such as setup/installer) harder in the short run, but much easier in the long run. However, we do also want to provide a neat package (for at least the major Linuxes, Mac and Windows) for the individual parts as well as the whole thing.
So you could download fpdb-full for Windows that will be a fully integrated package that will install fpdb, fpdb-hud, all the dependencies (such as python and mysql) where necessary, and will look and feel like one program from installation through setup to day-to-day use.

On more functional issues, I cannot think of a feature that I would reject. Many things I personally wouldn't be prepared to write (unless I'm being paid for it, of course) but if somebody sends a patch I'll probably take it, no matter how obscure the function. (Note: to not overload the interface some things will probably be turned off or hidden by default)

I think that fpdb has been getting neater and that we'll continue to try to make it as neat as we can. For example, I am working on combining the 2 config files right now as an exercise in neatening. That should be ready by the next release.

Are you asking us to have the installer put an icon on your desktop? That should be doable, and is exactly the neatening stuff we'll do.

Regarding fpdb and HUD being separate applications. When the config files are integrated, most users will not know that they are separate. We get nice performance advantages by separating the applications. Did you see the post above where someone is using fpdb to 14 table on an older laptop--try that when both applications are running in the same process. I think that having the two applications separate is a good design decision and maintains flexibility for later.

I think the discussion is about semantics really. You're talking about decoupling the hh format into an intermediate format to derive statistics, we are looking at decoupling the hh format into a stars like format[1] to derive statistics. The design difference is trivial, and easily fixed if it causes problems later.

The import and parsing code is going through a refactor at the moment, with one of the secondary goals being to make adding additional sites easy.

Steffen, Eratosthenes and myself all work under Linux, which currently "supports" Stars and FTP[2] so additional site support isn't likely to come from us any time soon. (Though i think Steffen will consider paid work to add support)

Given our general lack of ability to play other sites and generate hand histories, it is a goal to make external contributions and maintenance easier.

FYI, like Steffen I welcome any patches in this area. I'll test and apply patches pretty quickly. Check http://www.assembla.com/spaces/fpdboz/trac_git_tool if you want to help out, i'm syncing daily with Steffen and Eratosthenes

Sorrow.

[1] "Stars like" means it contains the site name, and may have a few extra sections for knockouts/bounties and other features that don't exist on stars.
[2] Out of the box using wine.

More experiences from the Pokerstars cash tables:

Again trying another session of 14–16 tables at once with the fpdb HUD today I observed an interesting effect, namely that the load on my system slowly grew as the session progressed. Eventually the processor load was permanently at 100%. I suppose I was sailing close to the wind by playing a two-hour session (towards the end the hand importer had to go through about 1400 hands on each cycle). Nevertheless both the Pokerstars client and the HUD windows remained responsive.

In spite of what my remarks earlier in the thread might have suggested, my experience has been that the hand parser is amazingly fast. And the output I get from “ps” (on Linux) confirms what Steffen (if I remember correctly) has said, viz. that most of the load is in fact borne by the MySQL process. (Result: 76 minutes of processor time compared with negligible values for fpdb.py and HUD_main.py.) So it may be that if there are gains in efficiency to be made it is in the interface with the database.

This is fantastic feedback Stranger.

I suspect that towards the end of your session MySQL is getting hammered by requests for "does this hand exist" as each update cycle re-reads every file and checks if every hand exists (1400 db queries to decide to process < 16 hands). So we may need to add some state to the auto-update(gui) parser to provide a non SQL lookup for previously parsed hands in a sesison.

Really appreciate the comments.

That sounds like a good idea. It’s nice to know my comments have been useful.

As further food for thought, the following elaborates on what I meant when I talked about efficiency in the interaction with the database:

At the moment the HUD cache is accessed three times whenever new values arrive. The hand importer first reads the current values and then it updates them. These values then have to be separately read from the database by the HUD process. Could these values instead be (a) kept locally in an array and (b) simply passed through the pipe to the HUD?

Another thing (although to an extent this is a purely selfish gripe) is that most of the information in the HUD cache is hardly ever needed by the actual HUD. I have set mine up to display just the following: hands seen, VPIP, PFR (pre-flop raise %) and 3-bet percentage; the rest doesn’t interest me. The HUD in general is only able to display six items of information, more than which would clutter the screen anyway. So perhaps for efficiency reasons there could be a second cache (of some sort) containing only the desired data. Anything else could simply be fetched as needed (e.g. by the pop-up window).

Hi E. L.-- I am glad you are using fpdb and posting about it here because you are our "worst case." Please keep it up.

I think if we improve our file reading algorithm in the import we will get a large improvement in efficiency. The first thing to do is to skip the text in the hh files that has already been imported.

sql has a way to easily increment fields in a table without first reading the table:

I have used that in the past, but don't remember enough of the details to know if this will help.

The database schema for fpdb defines keys and secondary keys which is good practice. This slows imports because mysql spends a lot of time looking for duplicate keys. This is why bulk imports significantly slow down when there are more than about 100k hands in the db.
When I was thinking about this particular version of the HUD one of the design options I thought about was having the importer code just pass the HUD stats information to the HUD through the pipe, as you suggested here. This would simplify things for the HUD and for a really simple HUD you could eliminate the database interface code completely. I decided to go with the present scheme because I wanted to keep the interface between the importing process and the HUD process simple and because I would need db interface code in the HUD for all but the most simplest HUDs anyway.

It would be possible only query the data we know we need from HudCache when we are updating the HUD. I think that the savings would be fairly small and would mean that we have to build our SQL on the fly. I have avoided building SQL in code to try to make things simpler and more reliable. Let's say you are playing 20 tables at about 80 hands/hr/table = 27 hands/min = 27 HudCache accesses/min. HudCache accesses from the HUD is not the problem--or only a tiny part of the problem. One (or more) of the closed-source HUDs doesn't update the HUD after every hand. It would be possible for us to take that approach, if we really need to.

It is barely possible that you could get better performance from postgres. (There is all sorts of conflicting lore about postgres v mysql performance under various loads and types of loads.) I think it would be a happy surprise if you did get significantly better performance from postgres, so I am not recommending that you drop everything and try it. It would be worth trying if we can't speed things up with what we are trying.

My 2 cents regarding parsers/converters. In my opinion it would be better to have a generic parser, not a converter. Create Parser interface (I don't think they have interfaces it Python) or abstract class, implement it in say PokerStarsParser. If another site has a hand history format similar to PS, then extend PokerStarsParser (or extract any common functionality to another super class) to reuse existing code. This way it would be easier to plug in new parsers for other sites. Ongame for example stores hand histories in SQLite database, so the parser implementation would be different. Of course all the parsers should produce hand histories in standardized internal format, which could then be stored in the database or whatever. I took a look at the code (it was a week ago though) and it didn't look quite possible to add another parser without making changes in lots of different places.

Nice project and thank you for doing that!

Eleatic Stranger --

We have made a change to the importer that should speed up your life. The import code now skips the hands that have already been imported. They are no longer parsed and checked to see if they are already in the db. This should speed things up a lot.

Steffen is out of touch until about Monday, so this won't be in an official release or the fpdb git repo until next week. You can get this now from my git repo here: git@git.assembla.com:free_poker_tools.git

You probably want to keep your hacked copy of the HUD and only use new fpdb files. Those would be:
fpdb*
Fpdb*
Gui*

So make a new working dir and a new dir to receive the git repo. Copy your working dir into the new working dir. Clone my repo into the new git repo file. Then copy the files above from the new git repo dir to your new working dir. (but you knew all this)

...crossing posting this in the different trackers...

This is a tournament hud request that I think would be popular. In the the tournament hud I would like to be able to display how many “Tournaments” I have played against each opponent and their “ITM %”. Heck even “Net Won” (not tournament chips won) and “ROI”. This would be like having a personal sharkscope of your opponents.

Thank you. I’ll try this and let you know how I find it.

I suppose it is all the SUM() expressions in the code involving the cache that have me a little worried. I have actually experimented with having the HUD updated less frequently than once per hand, with some improvement in speed.

The main reason for my worrying about performance at this point (given the immaturity of the project) is that I think because a heads-up display is constantly on the user’s screen it should be lightning-fast.

I am certainly your worst case (or worst nightmare) at the moment, but it is not terribly unusual for people to play 24 tables at once. When the project receives more attention and others try using the HUD on, say, 8 tables at once I think they are going to encounter problems and be disappointed. Of course, this remark might prove to be premature; I’ll report again once I have tried your new version.

As things stand, however, I do encounter sporadic instances of excessive slowness – in spite of what I have been saying about there being no errors! An HUD will occasionally need several seconds to appear, as I have discovered, which can be annoying. This means, for instance, that the HUD would not be usable on fast tables. But this doesn’t bother me; on my slow tables I am happy just to make the other players wait....

You see this delay when you click on the entry in the task bar? Then it takes a few sec for the HUD stat windows to actually appear?

Updating the gui of the HUD could be blocked by the HUD program waiting for the HudCache query to complete. The HUD and fpdb have their own db connections so the queries should be running in their own threads in the db, but your db is getting hammered and your cpu usage is high. I see a way to pull the HudCache query into a thread that is separate from the gui thread, this would make the HUD gui more responsive. This would allow even longer running queries without affecting the responsiveness. I need to think about this before I actually go do something stupid.

Yes, it does – sometimes. Allow me to elaborate.

First of all, many thanks for the new hand importer; it speeds things up enormously.

But there are still some problems with the HUD’s responsiveness, for quite interesting reasons.

One thing I didn’t mention is a hack I introduced earlier, and didn’t bother to get rid of, in an attempt to speed things up. I decided to see what would happen if I delayed the call to the database to collect the HUD cache information.

In HUD_main I changed the call to update the HUD:

to read:

making the appropriate changes to the parameters of the update method in Hud.py so that it could open its own connection to the database instead of having to use the one passed to it as a parameter, as was previously the case. I then changed the behaviour of the HUD so that it would only open this connection and demand the HUD information at such time as the user chose to make the HUD visible (since I keep most of the HUD windows invisible most of the time). This is very much a non-ideal solution.

I thought this modification of mine had probably done nothing to improve efficiency, and so changed it back after installing your new importer.

Unfortunately, what this now means is that opening a HUD becomes a sort of lottery: between hand-parsing cycles extremely responsive; but actually during a cycle unacceptably slow. By “unacceptably” I mean that it sometimes takes a HUD about ten seconds to show up at all. I know my machine is a bit slow, but even so!

My guess, however, is that there is an extremely simple solution. If I understand the code correctly, in Linux the HUD process is single-threaded. So perhaps all that is needed is a little multi-threading magic.

One of the things that is going on here is that when the python code (my code) makes a new stat window (or whatever) it doesn't actually show up on the screen until the python code is finished and it passes control back to the main_loop (that is, the gtk code). Therefore when the python code gets really busy it can be a while before the main_loop regains control and can update the screen. Because I have the HudCache query and the main_loop in the same thread, the main_loop cannot update during the relatively quiet time when the python code is waiting for a HudCache query. Clearly I need to drag everything I can out of the main_loop thread.

Have you lowered the interval between imports? I think that would help by dribbling the new hands in 1 or 2 at a time instead of flooding them in. The code that checks for new hands is reasonably efficient and will not take long when there are no new hands, so using something in the 1-3 sec range should be fine.

Hm. I don't currently know python, but I have been a programmer for a very long time, and I've been using a lot of slow languages (and computers!) that need a lot of speed optimizations, so I'm pretty good at that sort of thing.

Can someone point me to a post here, or if there isn't one, give me the very short form of what is needed to operate and develop for fpdb? (i'd dig through the thread, but i have to leave in like 1 minute to go play some live poker and it's like 500 replies)

There’s an idea; the problem may have an even simpler solution than I thought. I will let you know what happens.

I think this is worth pursuing, even if your suggestion to lower the interval produces a responsiveness which is “good enough”. It seems best to move the code dealing purely with the GUI into a different loop, since the desired behaviour is for it to display the information it currently has at fast as it can, meanwhile just letting new information filter in in its own time.

I think this is the post you want:

Python requires no special development tools since it is an interpreted language. Any text editor will do.

If you use windows, use the INSTALLER to set up your environment. If you use vista, the installer fouls up mysql (because of UAC), so install it before you run the installer. (There is a how to install on Vista on the mysql web site.) If you already have postgresql, you can use that.

Install git.

Pull source (or clone) from my git repo = git@git.assembla.com:free_poker_tools.git

WEB PAGE is here

This is true, but if you are an IDE kind of guy, Eclipse + pydev plugin is pretty nice. Runs well on windows and linux. You need a pc that is 2 yrs old or less probably.

I have tried lowering the interval for the automatic hand importer: still no luck. The symptoms are very strange. Sometimes an HUD will appear quickly, but then the data will disappear again for a few seconds for no readily apparent reason (even when it is obvious that there is no fresh hand from the affected table to be processed).

On the other hand, I cannot honestly be sure that these problems apply to the latest version of the code, as the code I am using is still a version of the old hacked source I was using originally (but incorporating Eratosthenes’ change to the parser to have it keep its place in the HH files).

But the following change to HUD_main seems to produce a big improvement:

In place of

this

with read_stdin slightly modified so that it goes on an infinite loop, waiting for something to appear on sys.stdin. I have just tested this by playing a few hands with very encouraging results. In the error log the following was reported

which I do not understand, but the program seemed to work (and speedily).

I am about to play a longer session to see what happens.

I should mention, since I feel I might be getting into deep waters here, that I know nothing about Python and I know nothing about threads (beyond knowing what they are). I imagine that this change brings with it various potential worries about thread safety, which to date I have not considered at all.

I spend some time today figuring out how to pull a whole bunch of code out of the main_loop (= gui) threadb without breaking windows|gtk|python. I will try to implement that tomorrow. This should make the gui a lot more responsive.

If this actually works, I'll give the stacked/cascaded tables thing a go. I have an idea or two on that now.

You have taken about half of the first step I'm planning on implementing in the morning. I am not sure why that doesn't crash--it might after you play a while.

hmm. I need a practically unheard of, outside of the linux hacker circle, version control system to get at the source .. I think I'll pass. I ditched nearly all open source software years ago, excluding subversion and the basic gnu toolset, and I'm not keen on going back to it. If it were a subversion repository, or there were regular diffs, that'd be something different - i've already got subversion, and it's practically mandatory for anyone in any kind of software development to use, it's practically an industry standard. I was willing to try to get Python actually functioning in Windows, but being forced to use something Linus designed/wrote is entirely too much .. particularly when there is no clearly well functioning version of it in windows, there's no gui implementation for windows, and windows command line sucks.