Two Plus Two Poker Forums - ** Python Support Thread **

Page 13 of 30

Show 100 post(s) from this thread on one page

Two Plus Two Poker Forums (https://forumserver.twoplustwo.com/)

- Computer and Technical Help (https://forumserver.twoplustwo.com/48/computer-technical-help/)

- - ** Python Support Thread ** (https://forumserver.twoplustwo.com/48/computer-technical-help/python-support-thread-1007515/)

Neko	11-13-2011 05:17 PM

Re: ** Python Support Thread **

Quote:

Originally Posted by calm (Post 29812186)

Yes I want option 2, but as a list comprehension instead of a for loop. Think I got it figured out thanks to Neko.

Here's another for loop I'm trying to substitute with a list comprehension:

Code:

row_data = []

row_color = []

for key in dict:

    row_data.append([])

    row_color.append([])

Now I could easily do this with two separate list comprehensions:

Code:

row_data = [[] for key in dict]

row_color = [[] for key in dict]

But since I'm then looping over the dict twice, I guess the for loop would be faster? Is there a way to combine the two in one list comprehension?

would definitely prefer the 2 line solution over 5 lines. You can use the timeit module to figure out which way is faster :)

Code:

import timeit



d = dict(zip(range(100),range(100)))



def method1():

    row_data = []

    row_color = []

    for key in d:

        row_data.append([])

        row_color.append([])

        

        

def method2():

    row_data = [[] for key in d]

    row_color = [[] for key in d]



t1 = timeit.Timer("method1()","from __main__ import method1,d")

t2 = timeit.Timer("method2()","from __main__ import method2,d")



print "t1"

print t1.timeit(10000)

print "t2"

print t2.timeit(10000)

output:

Code:

t1

2.03231943169

t2

0.910326269734

calm	11-13-2011 05:49 PM

Re: ** Python Support Thread **

Cool thanks. Haha I actually tried using timeit but got stuck an infinite loop somehow.

RoundTower

11-13-2011 06:26 PM

Re: ** Python Support Thread **

If you only want to do the dict comprehension once, you can do

Code:

row_data, row_color = zip(*(([],[]) for key in dic))

or if it's vitally important that row_data and row_color end up being lists instead of tuples,

Code:

row_data, row_color = map(list, zip(*(([],[]) for key in dic)))

If you think this is unreadable then just write it the other way, but it's good to know you have this option in case you do end up with this code in a performance bottleneck. zip(*spam) is a very useful idiom for transposing lists.

RoundTower

11-13-2011 06:55 PM

Re: ** Python Support Thread **

of course, if all you want is two lists of empty lists, each the same length as the dictionary, there are lots of efficient ways to do it. But the above is the generalizable version.

stanek

11-13-2011 07:02 PM

Re: ** Python Support Thread **

Code:

file = open("test.bin","wb")



for b in range(25209495):

    file.write("\xFF")

This writes a file of 25.2MB full of 1's

But the python process uses 388MB of RAM.

Can anyone explain why it is using so much memory and maybe suggest a more memory efficient code?

Neko	11-13-2011 07:11 PM

Re: ** Python Support Thread **

immediate thought is just change range to xrange (assuming your on python 2.X)

stanek

11-13-2011 07:19 PM

Re: ** Python Support Thread **

Quote:

Originally Posted by Neko (Post 29815171)

immediate thought is just change range to xrange (assuming your on python 2.X)

2.7.2

Now it uses 1.7 MB of RAM.

Thanks

Neko	11-13-2011 07:31 PM

Re: ** Python Support Thread **

The reason it used so much ram before is because range() creates the entire object in memory before iterating over it, while xrange uses a generator expression (http://docs.python.org/library/functions.html#xrange).

e i pi

11-13-2011 08:24 PM

Re: ** Python Support Thread **

I'm a newb trying to figure out the string format method for python 3. This is from http://learnpythonthehardway.org/book/ex24.html written for python 2.

Code:

def secret_formula(started):

    jelly_beans = started * 500

    jars = jelly_beans / 1000

    crates = jars / 100

    return jelly_beans, jars, crates



print "We'd have %d beans, %d jars, and %d crates." % secret_formula(start_point)

below doesn't work, giving "IndexError: tuple index out of range":

Code:

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))

but this does,

Code:

a,b,c = secret_formula(start_point)

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(a,b,c))

what gives?

RoundTower

11-13-2011 08:37 PM

Re: ** Python Support Thread **

secret_formula(start_point) is a tuple of three items (at least I assume it is). So you can use it either as one item or as three. But Python can't always tell which you mean.

When you try

Code:

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))

it interprets it as just one argument, which is the default.

When you do

Code:

a,b,c = secret_formula(start_point)

it is smart enough to realise that the tuple needs to be unpacked into three items which are then assigned to a, b and c.

To force it to unpack the tuple, you can use the * operator, like so:

Code:

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(*secret_formula(start_point)))

e i pi

11-13-2011 08:58 PM

Re: ** Python Support Thread **

merci

stanek

11-15-2011 02:15 AM

Re: ** Python Support Thread **

Optimization help please!

The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.

In the example below, we read each byte directly from the file into <byte_wiki>. This method takes approx 5-7 minutes and uses 1.8MB of RAM.

The commented out section reads the file before hand into a list and uses 690MB of ram and takes much longer to run, 9++ minutes.

nBytes = 25 million+

Code:

for b in xrange(1,nBytes+1):

    byte_copy = bin(random.randint(0,255))[2:].rjust(8,'0')

    #byte_wiki = bin(ord(source_wiki[b]))[2:].rjust(8,'0')

    byte_wiki = bin(ord(wiki7z.read(1)))[2:].rjust(8,'0')

    

    list_copy = []

    list_wiki = []

    for bit in byte_copy:

        list_copy.append(bit)

    for bit in byte_wiki:

        list_wiki.append(bit)

    for compare in xrange(8):

        if list_copy[compare] == list_wiki[compare]:

            score += 1

Why is this code so slow?

I am guessing that when the code is read from file that i/o is a bottleneck.
I am guessing that when fed into a list that the 25+million elements are what is slowing it down.

My guess is that splitting the Hex byte values and formatting them into lists, and then comparing each bit are expensive operations.

Also having list_copy = [] and list_wiki = [] within a loop looks silly to me but I couldn't think of another way around that.

Looks like I should probably learn how to do some bitwise manipulation tomorrow.

Xhad	11-15-2011 02:44 AM

Re: ** Python Support Thread **

I'm rusty and not 100% that this is what you're trying to do, but it looks like you can dispense with all this list stuff and just XOR them...

Code:

for b in xrange(1,nBytes+1):

    byte_copy = random.randint(0,255)

    byte_wiki = ord(wiki7z.read(1))

    

    score += bin(byte_copy ^ byte_wiki).rjust(9, '0').count('0') - 1

EDIT: The "-1" is because I think it counts the '0b' at the beginning of the string form of the binary number and you don't want that. And I had to add back in the leading 0 padding. This will put the leading zeros in the wrong order but I don't think it matters since you will never use this string again from the looks of it. ;)

EDIT2: Actually now that I think about it some more I think this works even better:

Code:

score += 8 - bin(byte_copy ^ byte_wiki).count('1')

Again, I'm rusty and actually just muddling through this for practice

TheIrishThug

11-15-2011 07:58 AM

Re: ** Python Support Thread **

You can use izip to iterate over the two lists at the same time. This way you don't need to do the copy loop followed by the compare loop.

tyler_cracker

11-15-2011 11:53 AM

Re: ** Python Support Thread **

what are you actually trying to do?

Quote:

The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.

i have a hard time thinking of why you would need to do this rather than just use 'cmp' to tell you if the files are the same or not. i guarantee 'cmp' will be faster than any python program you could ever write (for all but the most degenerate cases).

e i pi

11-16-2011 01:45 AM

Re: ** Python Support Thread **

Code:

b = [1,2,3,4,5,6,7]

for g in b:

    print(b.pop(0))

this only returns 1-4 when run. why?

calm	11-16-2011 01:54 AM

Re: ** Python Support Thread **

Quote:

Originally Posted by e i pi (Post 29867144)

Code:

b = [1,2,3,4,5,6,7]

for g in b:

    print(b.pop(0))

this only returns 1-4 when run. why?

I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.

Try:

Code:

for g in b:

    print g

or:

Code:

while b:

    print b.pop(0)

e i pi

11-16-2011 02:35 AM

Re: ** Python Support Thread **

Quote:

Originally Posted by calm (Post 29867237)

I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.

That makes perfect sense, thanks.

Just playing around with this, python returns about half the list no matter the size and raises no errors.

Xhad	11-16-2011 03:03 AM

Re: ** Python Support Thread **

oddly enough, if you make it "for i in range(len(b))" it works properly. I guess the "for x in b" construct makes it so that when you pop and iterate, whatever counting method it's using ends up moving two spots instead of one.

daveT

11-26-2011 08:47 PM

Re: ** Python Support Thread **

Using Python 3.

I can't figure out why r returns as None:

This is one of those things where I know I should know the answer, but for some reason....

Code:

def fac(t, r = 1):

    r *= t[-1]

    if len(t) == 1:

        return r

    fac(t[:-1], r)



def fun20():

    t = []

    for i in range(1, 101):

        t.append(i)

    a = fac(t)

    b = sum(list(map(int, str(a))))

    print(b)

breaks

while this will print out the answer:

Code:

def fac(t = t, r = 1):

    r *= t[-1]

    if len(t) == 1:

        print(sum(list(map(int, str(r)))))

    fac(t[:-1], r)

I'll torture you guys to add that I know this could be done in one line of code, but I decided to do this one the hard way because I wanted to be sure I can do a factorial without busting, which apparently, I can't.

Xhad	11-26-2011 09:51 PM

Re: ** Python Support Thread **

You forgot to put "return" at the end

daveT

11-26-2011 10:04 PM

Re: ** Python Support Thread **

<headslap.gif>

Xhad	11-26-2011 10:53 PM

Re: ** Python Support Thread **

if it makes you feel any better I literally just made the same mistake

e i pi

11-28-2011 12:14 PM

Re: ** Python Support Thread **

So I've been casually chugging along learn python the hard way until this,

http://learnpythonthehardway.org/book/ex46.html

where the author basically says download a bunch of testing and installation packages and figure out how they work on your own :( wtf! all the documentation is for linux also but I'm using windows.

I finally figured out how to install the packages but I can't get this nosetests thing to work. I've found this file, C:\python32\scripts\nosetests-script.py, that runs but it doesn't actually test anything. Do I have to pass in the directory of my "project skeleton" as an argument or something?

Also is any of this stuff necessary? or is there an easier way to do testing that doesn't require using the command prompt like its 1996? I would gladly give up on trying to figure this out at this point if this stuff is out dated or just unnecessary.

Neko	11-28-2011 09:11 PM

Re: ** Python Support Thread **

Normally you just have to do:

Code:

C:\Path\To\Your\Project> nosetests.exe .

Nose is an excellent testing framework. Your statement about command lines being out of date is misguided fwiw. The default windows shell is terribad but understanding how to use a command prompt will be a major productivity booster.

All times are GMT -4. The time now is 04:26 PM.

Page 13 of 30

First

3 8 9 10 11 12 13 14 15 16 17 18 23

Last

Show 100 post(s) from this thread on one page