Free script to Datamine UB - Page 2 - Poker Software -- Two Plus Two Poker Forums

Willy, try opening the file using the command prompt, and you will be able to see the error message before it just shuts down. FWIW I had more success editing the file in notepad than the Python gui. Some syntax problem made it pack up before it got started.

Cheers all

Quote

11-28-2008 , 08:11 PM

#27

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Hey Willy,
Try this:

Quote:

try is to open up a command prompt. Change to the directory that the script is in and try to run it from the command line.
Start->Run
type cmd then enter.
change directories to where the script is e.g.
C:\ cd C:\Program Files\ub_mine <enter>
Then try running like so:
C:\ C:\Python26\python.exe ub_mine.py

and tell me what the error says.

fwiw from what I understand there has been some fairly major changes in the language for version 3.0 so it may be something to do with that.

Quote

11-28-2008 , 08:36 PM

#28

AP_Villain

grinder

Join Date: Dec 2007 Posts: 566

Quote:

Originally Posted by WillyT

I'm using the last code posted here and have changed the directory name in the code for use with vista.

What am I missing to get this working. When I double click the .py file the cmd prompt comes up briefly and then disappears and there's not sign of anything going on. The 'histories' folder has never been created.

Probably something I'm overlook having to do with Vista. Any ideas?

Thanks,

Bill

What is "directory name in the code" change for vista?

Quote

11-28-2008 , 08:40 PM

#29

AP_Villain

grinder

Join Date: Dec 2007 Posts: 566

Quote:

Originally Posted by Neko

Hey Willy,
Try this:

and tell me what the error says.

fwiw from what I understand there has been some fairly major changes in the language for version 3.0 so it may be something to do with that.

C:\Users\Scott>cd c:\users\scott\documents\poker\ubmine

c:\Users\Scott\Documents\Poker\UBmine>c:\python26\ python.exe ubmine.py
Traceback (most recent call last):
File "ubmine.py", line 96, in <module>
main()
File "ubmine.py", line 22, in main
os.chdir(replay_dir)
WindowsError: [Error 2] The system cannot find the file specified: 'C:\\Program
Files\\UltimateBet\\InstantReplay'

c:\Users\Scott\Documents\Poker\UBmine>

Quote

11-28-2008 , 08:41 PM

#30

AP_Villain

grinder

Join Date: Dec 2007 Posts: 566

The original script worked fine for me, although I haven't run it long enough to see the duplicate problem

The new script posted on this page is producing this error

Quote

11-28-2008 , 08:46 PM

#31

AP_Villain

grinder

Join Date: Dec 2007 Posts: 566

hrm now when i try to do the original, it says that it is not a valid win32 app

maybe i just need to reboot

Quote

11-28-2008 , 08:50 PM

#32

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

ap_villan you probably just need to change the replay_dir back to :

Code:

#location of UB instant replay data files
replay_dir = "C:\Poker Application\UltimateBet\InstantReplay"

All, I will add more error handling to the script tomorrow a.m. so that it handles problems more gracefully and at least doesn't crash without a more helpful error message.

Quote

11-29-2008 , 01:20 AM

#33

Pwoita

journeyman

Join Date: Aug 2008 Posts: 344

Any chance to check if file is over a certain size like 750k, then if so, rename to converted_1.txt, converted_2.txt, etc and create new converted.txt file ... just so the file doesn't get too large?

Also, is there any need to keep the .dat files in the converted directory?

Quote

11-29-2008 , 07:05 AM

#34

WillyT

veteran

Join Date: Jun 2005 Posts: 2,301

Got it working now TYVM.

In Vista I had to set python to run as administrator and to change the text file with the script in it to a python file in the cmd prompt.

Cheers,

Bill

Quote

11-29-2008 , 10:11 AM

#35

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Quote:

Originally Posted by Pwoita

Sure I can set a limit on either the size of the file or number of hands stored in one hand.

I will make it an option whether you want the converted.dat files moved to the converted directory or just delete them once they've been converted.

Glad to hear you've got it working Bill

Quote

11-29-2008 , 04:55 PM

#36

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Here is a substantially improved version of the script. There is now a cap on the number of hands written into 1 file which you can set at the top of the file (default is 100 hands). The script checks both the XP and Vista directories at the start so hopefully it will work for everyone out of the box. There is more error handling code now, so it should never crash without at least giving you an error message before it dies.

Let me know if you have problems or feature requests.

Procedure for running is same as before.

Code:

#ub_mine.py v1.1 Nov 29 2008
#written by Neko (2+2 handle)
#This code is released under the WTF licence. That means you can
#do W.hatever T.he F.uck you feel like with it.
#If you would like to make a small donation via UB or Absolute p.m. me on 2+2
#or email me neko2p2@gmail.com

import os,shutil,sys,string,time,traceback
from msvcrt import kbhit,getch

#maximum number of hands to store in one batch file
max_hands_per_file = 100

#if True converted .dat files will be deleted otherwise they will be moved
del_converted = True

#list of locations to look for UB instant replay data files
replay_dirs = ["C:\Program Files\UltimateBet\InstantReplay",
               "C:\Poker Application\UltimateBet\InstantReplay"]

#move instant replay files to this directory after processing
root_dir = os.getcwd()
converted_dir = root_dir+"/converted/"

#store the converted histories in this directory
hist_dir = root_dir+"/histories/"

#number of seconds to wait in between processing batches
sleep_time = 5

#===============================================================================
def main():
    replay_dir = get_replay_dir(replay_dirs)
    if not replay_dir:
        die("Unable to find the instant replay directory")
    
    #make directory to move processed hands to if it doesn't already exist
    print "Looking for processed hand directory "+converted_dir
    if not os.path.isdir(converted_dir):
        "\tConverted directory not found. Trying to create now...",        
        try:
            os.mkdir(converted_dir)
        except:
            die("failed")
        else:
            print "success"

    else:
        print "\tAlready exists"
        
    #make directory to store converted hand history file
    if not os.path.isdir(hist_dir):
        try:
            os.mkdir(hist_dir)
        except:
            die("failed to make hand history directory")

    stop = False

    print "="*80
    print "Starting to convert hand histories"
    print "hit 'q' at any time to stop import"
    print "="*80

    ttl_hand_count = 0
    hand_counter=0
    out_file = None
    os.chdir(replay_dir)

    while not stop:

        if kbhit() and getch() == 'q': stop = True
        
        dirList=os.listdir(os.getcwd())

        for fname in dirList:
            if fname.find('.dat')!=-1:

                #get a new file to write to if the old one already contains
                #the max num of histories or an output file doesn't yet exist
                if hand_counter ==  max_hands_per_file or not out_file:
                    hand_counter = 0
                    if isinstance(out_file,file):
                        out_file.close()

                    out_file = get_new_file_object()

                #extrat history and move/delete the .dat file
                if add_to_file(out_file,fname):
                    
                    handle_converted_file(fname)
        
                    hand_counter += 1
                    ttl_hand_count += 1
                    sys.stdout.write("\rTotal Hands Written: " + str(ttl_hand_count)) 
                    sys.stdout.flush()
           

        if not stop: time.sleep(sleep_time)

    if out_file:
        out_file.close()
    raw_input("\nSuccesfully Finished Execution. Press enter key to quit.")

#-------------------------------------------------------------------------------
#Extract the hh from the .dat file
def add_to_file(f,fname):
    try:
        infile = open(fname,'r')
        
        dat = infile.readlines()
        for i,line in enumerate(dat):
            dat[i] = dat[i].strip()
        infile.close()
    
        first = dat.index("[IHH]")
        dat = dat[first+1:len(dat)-1]
        hh = ''
        for line in dat:
            nxt_line = line.partition('=')[2]
            hh += nxt_line+'\n'
    
        f.write(hh)
        f.flush()
    except:
        return False

    return True

#-------------------------------------------------------------------------------
#remove or delete the converted .dat files
def handle_converted_file(fname):
    if del_converted:
        os.remove(fname)
    else:
        # removing duplicate file if file exists
        pathToFile = converted_dir+fname
        if os.path.exists(pathToFile):
            os.remove(pathToFile)
        shutil.move(fname,converted_dir)

#-------------------------------------------------------------------------------
#iterate over the list of replay directories and return the first one that exists
def get_replay_dir(dirs):
    idx = 0
    print "Looking for Instant Replay directory...."
    for rdir in dirs:
        print "\tChecking "+rdir,
        if os.path.isdir(rdir):
            print "found"
            return rdir
        else:
            print "doesn't exist"

    return False
#-------------------------------------------------------------------------------
#generic exit point on errors
def die(msg = None):
    if msg: print msg
    raw_input("Press enter key to exit")
    os._exit(99)
#-------------------------------------------------------------------------------
#returns the next batch file name
def get_output_file_name():

    suf = get_file_suffix()
    num=suf.next()
    while os.path.exists(hist_dir+"batch_"+str(num)+'.txt'):
        num  = suf.next()

    return hist_dir+"batch_"+str(num)+'.txt'

#-------------------------------------------------------------------------------
#batch file suffix generator
def get_file_suffix():
    k = 0
    while True:
        k+=1
        yield k
#-------------------------------------------------------------------------------
#returns a new file object to write a batch of histories to
def get_new_file_object():
           
    try:
        out_file = open(get_output_file_name(),'w')
        return out_file
    except:
        die("Unable to create new output file "+fname)

#===============================================================================
if __name__ == '__main__': 
    try:
        main()
    except:
        print ""
        print "An unhandled exception occured."
        traceback.print_exc()
        raw_input("press enter key to exit")

Quote

11-29-2008 , 08:57 PM

#37

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

has anybody tried the latest version yet? I've had it running for a few hours and recorded 2000+ hands without trouble so far.

Quote

11-30-2008 , 02:56 AM

#38

Pwoita

journeyman

Join Date: Aug 2008 Posts: 344

Workikng well for me. Thanks man.

Btw ... you could rename functions to the comments, so you don't need as many comments ... and the code becomes self-commenting. Eg.
add_to_file => extract_and_write_to_file
handle_converted_file => remove_converted_dat_files
Makes it easier for you to read and modify if you need to change it in a year after not looking at it and needing to figure where to change something.
Not a criticism ... just a though for your own coding since I have had to look at my old code before and wish I had done it like that so it was less work for me in the future

Thanks again for the script.

Quote

11-30-2008 , 10:49 AM

#39

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

haha yeah understood. Was just trying to bang it out as fast as possible.

Quote

01-06-2009 , 10:06 AM

#40

feint06

adept

Join Date: Jun 2007 Posts: 969

I just changed the replay directories and it works on Absolute Poker...

Code:

#AP_mine.py v1.1 Jan 05 2009
#Mostly written by Neko (2+2 handle)
#This code is released under the WTF licence. That means you can
#do W.hatever T.he F.uck you feel like with it.
#If you would like to make a small donation via UB or Absolute p.m. me on 2+2
#or email me neko2p2@gmail.com

import os,shutil,sys,string,time,traceback
from msvcrt import kbhit,getch

#maximum number of hands to store in one batch file
max_hands_per_file = 100

#if True converted .dat files will be deleted otherwise they will be moved
del_converted = True

#list of locations to look for AP instant replay data files
replay_dirs = ["C:\Program Files\Absolute Poker\InstantReplay",
               "C:\Poker Application\Absolute Poker\InstantReplay"]

#move instant replay files to this directory after processing
root_dir = os.getcwd()
converted_dir = root_dir+"/converted/"

#store the converted histories in this directory
hist_dir = root_dir+"/histories/"

#number of seconds to wait in between processing batches
sleep_time = 5

#===============================================================================
def main():
    replay_dir = get_replay_dir(replay_dirs)
    if not replay_dir:
        die("Unable to find the instant replay directory")
    
    #make directory to move processed hands to if it doesn't already exist
    print "Looking for processed hand directory "+converted_dir
    if not os.path.isdir(converted_dir):
        "\tConverted directory not found. Trying to create now...",        
        try:
            os.mkdir(converted_dir)
        except:
            die("failed")
        else:
            print "success"

    else:
        print "\tAlready exists"
        
    #make directory to store converted hand history file
    if not os.path.isdir(hist_dir):
        try:
            os.mkdir(hist_dir)
        except:
            die("failed to make hand history directory")

    stop = False

    print "="*80
    print "Starting to convert hand histories"
    print "hit 'q' at any time to stop import"
    print "="*80

    ttl_hand_count = 0
    hand_counter=0
    out_file = None
    os.chdir(replay_dir)

    while not stop:

        if kbhit() and getch() == 'q': stop = True
        
        dirList=os.listdir(os.getcwd())

        for fname in dirList:
            if fname.find('.dat')!=-1:

                #get a new file to write to if the old one already contains
                #the max num of histories or an output file doesn't yet exist
                if hand_counter ==  max_hands_per_file or not out_file:
                    hand_counter = 0
                    if isinstance(out_file,file):
                        out_file.close()

                    out_file = get_new_file_object()

                #extrat history and move/delete the .dat file
                if add_to_file(out_file,fname):
                    
                    handle_converted_file(fname)
        
                    hand_counter += 1
                    ttl_hand_count += 1
                    sys.stdout.write("\rTotal Hands Written: " + str(ttl_hand_count)) 
                    sys.stdout.flush()
           

        if not stop: time.sleep(sleep_time)

    if out_file:
        out_file.close()
    raw_input("\nSuccesfully Finished Execution. Press enter key to quit.")

#-------------------------------------------------------------------------------
#Extract the hh from the .dat file
def add_to_file(f,fname):
    try:
        infile = open(fname,'r')
        
        dat = infile.readlines()
        for i,line in enumerate(dat):
            dat[i] = dat[i].strip()
        infile.close()
    
        first = dat.index("[IHH]")
        dat = dat[first+1:len(dat)-1]
        hh = ''
        for line in dat:
            nxt_line = line.partition('=')[2]
            hh += nxt_line+'\n'
    
        f.write(hh)
        f.flush()
    except:
        return False

    return True

#-------------------------------------------------------------------------------
#remove or delete the converted .dat files
def handle_converted_file(fname):
    if del_converted:
        os.remove(fname)
    else:
        # removing duplicate file if file exists
        pathToFile = converted_dir+fname
        if os.path.exists(pathToFile):
            os.remove(pathToFile)
        shutil.move(fname,converted_dir)

#-------------------------------------------------------------------------------
#iterate over the list of replay directories and return the first one that exists
def get_replay_dir(dirs):
    idx = 0
    print "Looking for Instant Replay directory...."
    for rdir in dirs:
        print "\tChecking "+rdir,
        if os.path.isdir(rdir):

            print "found"
            return rdir
        else:
            print "doesn't exist"

    return False
#-------------------------------------------------------------------------------
#generic exit point on errors
def die(msg = None):
    if msg: print msg
    raw_input("Press enter key to exit")
    os._exit(99)
#-------------------------------------------------------------------------------
#returns the next batch file name
def get_output_file_name():

    suf = get_file_suffix()
    num=suf.next()
    while os.path.exists(hist_dir+"batch_"+str(num)+'.txt'):
        num  = suf.next()

    return hist_dir+"batch_"+str(num)+'.txt'

#-------------------------------------------------------------------------------
#batch file suffix generator
def get_file_suffix():
    k = 0
    while True:
        k+=1
        yield k
#-------------------------------------------------------------------------------
#returns a new file object to write a batch of histories to
def get_new_file_object():
           
    try:
        out_file = open(get_output_file_name(),'w')
        return out_file
    except:
        die("Unable to create new output file "+fname)

#===============================================================================
if __name__ == '__main__': 
    try:
        main()
    except:
        print ""
        print "An unhandled exception occured."
        traceback.print_exc()
        raw_input("press enter key to exit")

Quote

01-11-2009 , 07:21 AM

#41

the clêaner

veteran

Join Date: Aug 2006 Posts: 2,704

this is awesome. thx

Quote

01-12-2009 , 06:48 PM

#42

wernerw

newbie

Join Date: Apr 2006 Posts: 18

Is there an exe out by now? Would love to run it and not bother to install Python if not necessary.

Is there any tool out that opens windows at ub/Abs?

Quote

01-12-2009 , 09:25 PM

#43

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

I just made an executable for UB_mine using bbfreeze which you can download here:

http://www.mediafire.com/?sharekey=e...db6fb9a8902bda

Just extract the zip file in your C:\Program Files directory and then you should be able to double click the UB_mine.exe file to get up and running. I have only tested this on 32 bit Vista but I imagine it will work on XP too.

There was a tool posted in this forum by Tomatoes a little while ago that will do fully automated data mining on UB for $25 I think.

Quote

01-13-2009 , 12:00 AM

#44

loco

The First Golden

Join Date: Jul 2008 Posts: 14,846

So siiiiiiiiiiiiiiiiiiiiiiick!!!! I read your post and 6 minutes later I was mining Cereus!!!! WTF?!!!! PM me your UB screename dude, I'll ship you 10 bones.

Quote

01-13-2009 , 02:12 PM

#45

wernerw

newbie

Join Date: Apr 2006 Posts: 18

I run Vista and cannot start it because the "side by side configuration is invalid", whatever this means :-)

Quote

01-13-2009 , 04:11 PM

#46

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Hmmm. Not sure what that is and google isn't really turning up anything to useful. Is it 64 bit vista? Have you installed vista service pack 1 yet? Do you have User Acces Control enabled?

Doing some more googling I found http://rant.blackapache.net/2008/05/...ration-issues/ which might be helpful (read comments). Also check out this http://babyface.name/2008/02/10/appl...-is-incorrect/

Quote

01-15-2009 , 07:14 AM

#47

wernerw

newbie

Join Date: Apr 2006 Posts: 18

Quote:

Originally Posted by Neko

32-bit Vista, SP1 installed. Well, maybe I try to install python and run the script there. I am lost

Quote

01-15-2009 , 10:27 AM

#48

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

You might want to try installing this package from Microsoft to see if it will work after http://www.microsoft.com/downloads/d...displaylang=en

If not installing python is very straightforward. Just make sure you download python v2.6 because I don't think this script will work with v3.0

Quote

01-15-2009 , 10:35 AM

#49

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Also for anyone who uses this script: I am doing a 200 km (125 miles) charity bicycle ride to raise money for cancer research ( http://to09.conquercancer.ca/site/Pa...=to09_homepage ) and need to raise $2500 to enter. Please consider donating a few bucks if you find this script useful. I have already received a few donations for this script and have decided that I will be putting them towards my fundraising effort.

If you PM me I can send you a link where you can donate online with a credit card or I can take a site transfer and make the donation myself.

Sorry for the spam, but I think it is for a good cause.

Quote

01-18-2009 , 10:53 AM

#50

wernerw

newbie

Join Date: Apr 2006 Posts: 18

I installed Python, working perfectly, thank you.

I wonder why you only allow 100 hands per file. Is there a performance issue if you raise that number? I would like to raise it to 500 or 1000, so I do not have so many files.

Quote

Page 2 of 8

First

1 2 3 4 5 6 7

Last

Post Reply Subscribe

...

Page 2 of 8

First

1 2 3 4 5 6 7

Last