Hi Dog,

I meant to reply to this earlier, sorry for the gap ...

I dunno mate, I only have a fairly small db (say generally 20k-30k hands) and I only have 80k or 90k hands all time (if I even have the histories for all of them). If I make any rash comments about which db to prefer, you have to understand it's hunch and bias on my part as much as anything.

But I do suspect postgres is better at the serious size db's, so you might find a difference ... but having said that I think management is the key thing when you have that many hands. I think I would have several db's; a main db for playing (but only say 100k hands or less), and then maybe a series of 100k dbs for history. Then try out a small db occasionally when new code comes out.
Note that the new importer is definitely quicker than the old one - but then I dunno how much it will slow down on your size db ... but for 5-figure db sizes I think 50+ hands/second is possible/likely.

For the config file, I think you can just run with an old one generally, I've only just updated mine - you're right it's a pain

- sc

Might be nice to support multiple database elements in the config, and select between them within the GUI.

You can define multiple dbs in the config, like this:

Then specify the one you want to use with the -d option on the command line. (you actually set up shortcuts and click on the one you want, eh?)

Reimporting is going to be necessary if you want to keep up with the latest code. We do talk about these things and try to not break backwards compatibility for no reason.

People who have a lot of hands might split them up in to a few dirs and just import the most recent hands the first night.

When we add a new option to the config we usually supply a sensible default. Therefore, you can always try your old config and see if it breaks.

Thanks, this is awesome!

Thanks for the input. I had postgres and hated it, but some people like hot dogs and some hamburgers... I think I'll stick with Mysql for now. I am fairly organized on the HH's. Every night I zip them up and store them by quarters. I probably just need, as you suggested, to use mainly the newest quarter. Probably more accurate on players since players come and go, and often either get worse or better.

I'm going to try and install the newest update, then recreate the db and just see how long it does take. I'll post the results here. Thanks again.

Dog

There will be a new snapshot "soon". You'll want it, because it has some good bug fixes in it. If reimporting is painful, you should probably wait and see if it pops up.

Please say you didn't say that!!!

I installed the new update and am now in the process (approaching the one hour mark) of just the recreating the db. I still have to import all the hands. Too late to stop now. I may reduce the amount of bulk import files. I just looked... I have almost 2000 files, 950megs of hh's.

Dog

test1.py and test2.py both exist in the pyfpdb/ directory of the latest snapshot.

I recommend supplying a user and password that only has access to that database you created, or there is potential (however unlikely) that fpdb could access/conflict with the HEM/PT3 database.

Sorrow.

It's now been two hours since I clicked "Create or Recreate Database" and it is STILL chomping away, so it's not locked. Damn.. how long can it take to delete, where's the delete key...

Dog

Purely on a theoretical basis one can predict that both MySQL and Postgres will cope easily with very large data sets (e.g. millions of hands). Databases are designed for this.

P.S. I've got close to 1 Gb in hand histories, so I may try to replicate Dog's experiment (as soon as I upgrade to a more powerful computer).

I have PostgreSQL8.3 from having pt3..

i cannot put anything or understand how to get anything into there to allow access or anything to that sort. i am clueless.

Dog,

I think I have pushed about 300k in it at once, just when messing around. Rebuild your OWN Hudcache down to a significantly smaller period of time, and your speeds should remain just fine. As far as I can tell, everything seems to remain pretty linear as far as speed of queries, except for inserts into the hudcache - if you're inserting into a record for someone you've got 200k hands on (that's .. just you, because you haven't played all of your hands with the same people, i'd hope) it takes a while to update that. You definitely don't need a year or more of history in your hud cache for yourself (and probably if you have 100k hands on any other players, you don't need that whole set in your hud cache).

I'm not personally seeing any speed difference at all between the new import and old import, except that there seems to be less of a difference between the hand import speed on hands where "nothing happened" vs "everyone at the table exploded in a giant cloud of chips". However, I haven't actually run any large bulk imports on it. (I do have most of a year and a half of handhistories archived around here somewhere's that I could and do intend to throw at it, once the numbers are in better shape)

it'll go much faster if you login to mysql, give it the command "drop database fpdb;" then give it the create command again (look at the fpdb mysql install instructions for specific syntax) When you tell it to recreate a table, it does all sorts of junk that isn't really necessary for what you actually want to achieve

That code needs to be revisited I think.

This is much quicker, but requires root or database create permissions on the server.

PostSQL8.3 keeps asking for a password that i never set, and i cant create one.. ?? wtf!?

even downloaded and installed ALL of that stuff.. still cant find the PROGRAM to launch the installer from, or HOW to do it? please help someone?

I have no idea what you are attempting to do. I suspect you are stuck trying to unzip the snapshot. The snapshots have no installer.

If you have the environment installed using the file above, you need to download the latest snapshot, extract/unzip it to a folder (eg. FOLDER) and double click the file FOLDER/pyfpdb/fpdb.py

This will get you to the point where you need to enter details into the config file.

Sorrow

fpdb python file 20kb: heres what happens when i attempt to open it..

a black python screen pops up, then disappears in 2 seconds, cant even read it..

everytime i try and open a python file its creating a new one at the bottom 'name' compiled python file..

think they are error reports, even when i open them they close asap

perhaps python is not connected to postgresql8.3 that i have and thats the root of the errors ?

I'm just about to release another snapshot - it contains a file called test2.py that you can double click, which will tell us a bit more about what is working in your environment.

Sorrow

New snapshot time.


Lots of patches and bugfixes in this release.


If you haven't read it yet please take a look the release notes for the previous snapshot - http://forumserver.twoplustwo.com/sh...postcount=2485



Notes for 0.12-20100205 - tarball and .deb available in the snapshots directory on sourceforge.

• Fixes for utf encoding and display
• Graphing - gametype selection now works (ie. holdem or omaha etc)
• Fix HUD for tournaments
• Fix HUD for Stud
• Fix auto-import for FTP - Python UTF16 parsing when using seek() is broken.
• PartyPoker: Fixed limit and freeroll support for tourney's
• PlayerStats/Grapher: Autocomplete Hero names only lists the valid names for that site.
• Full Tilt: Player name parsing. Previous versions were matching too greedily resulting in erroneous playernames like 'sorrow (button)'
• Carbon Poker support: Alpha support. Cash games (NLHE), and bulk import only at the moment. Need someone who can actually run the client before we can implement and test hud functionality. We could use further hand history files specifically for different games here. Patch from Matthew Boss
• Everleaf Tournament support - NLHE
• Make import timer work again
• Fix double count bug when importing duplicates
• Allow Stars tablename to include #
• Log rotation, should now limit log files to 5x100Mb files.
• Allow Stats and Graph page 'left pane' be refreshed - ie. after you import a new game or stake.
• Move disabled sites to the bottom of converter list (1 site enabled, means the selection is always right)

Special props to Matthew Boss - a long time user who wanted to contribute something back, so wrote the converter for Carbon poker, a site he doesn't play on.

May variance be with you.

Sorrow

The snapshots doesnt contain any zip-files? Tar.gz doesnt work for windows if you dont install 3rd party software.

EDIT: I assume that tar.gz is some hardcore programmer zipmethod that is technically better then zip/rar but the best packing-method is not the one with the best packing, is the one easiest to use (that we all have and that needs no 3rd party installing). I hate when people use strange formats for images like .DAA when there is .ISO and .BIN that is compatible with everything.

so when i first opened it after the new snapshot i got that i didnt have a program.. then i downloaded that and clicking the FPDB.py is actually bringing up the program!!

Now what do i do from here? all it has up is a help tab telling me about a few changes you've made.. lol, i feel like a ******.

Use

http://www.7-zip.org/

to decompress.

Here is a megaupload of a zipped version instead since nobody uses 7zip.

http://www.megaupload.com/?d=FJDQYJHX