Open Side Menu Go to the Top

11-13-2011 , 05:17 PM
Quote:
Originally Posted by calm
Yes I want option 2, but as a list comprehension instead of a for loop. Think I got it figured out thanks to Neko.

Here's another for loop I'm trying to substitute with a list comprehension:
Code:
row_data = []
row_color = []
for key in dict:
    row_data.append([])
    row_color.append([])
Now I could easily do this with two separate list comprehensions:
Code:
row_data = [[] for key in dict]
row_color = [[] for key in dict]
But since I'm then looping over the dict twice, I guess the for loop would be faster? Is there a way to combine the two in one list comprehension?
would definitely prefer the 2 line solution over 5 lines. You can use the timeit module to figure out which way is faster

Code:
import timeit

d = dict(zip(range(100),range(100)))

def method1():
    row_data = []
    row_color = []
    for key in d:
        row_data.append([])
        row_color.append([])
        
        
def method2():
    row_data = [[] for key in d]
    row_color = [[] for key in d]

t1 = timeit.Timer("method1()","from __main__ import method1,d")
t2 = timeit.Timer("method2()","from __main__ import method2,d")

print "t1"
print t1.timeit(10000)
print "t2"
print t2.timeit(10000)
output:

Code:
t1
2.03231943169
t2
0.910326269734
** Python Support Thread ** Quote
** Python Support Thread **
150% up to $2,000 Welcome Bonus on CoinPoker
Join the action now
Daily Rewards • Splash Pots • CoinRaces
** Python Support Thread **
11-13-2011 , 05:49 PM
Cool thanks. Haha I actually tried using timeit but got stuck an infinite loop somehow.

Last edited by calm; 11-13-2011 at 05:52 PM. Reason: Oh I see it defaults to one million trials lol, that was the problem
** Python Support Thread ** Quote
11-13-2011 , 06:26 PM
If you only want to do the dict comprehension once, you can do

Code:
row_data, row_color = zip(*(([],[]) for key in dic))
or if it's vitally important that row_data and row_color end up being lists instead of tuples,

Code:
row_data, row_color = map(list, zip(*(([],[]) for key in dic)))
If you think this is unreadable then just write it the other way, but it's good to know you have this option in case you do end up with this code in a performance bottleneck. zip(*spam) is a very useful idiom for transposing lists.
** Python Support Thread ** Quote
11-13-2011 , 06:55 PM
of course, if all you want is two lists of empty lists, each the same length as the dictionary, there are lots of efficient ways to do it. But the above is the generalizable version.
** Python Support Thread ** Quote
11-13-2011 , 07:02 PM
Code:
file = open("test.bin","wb")

for b in range(25209495):
    file.write("\xFF")
This writes a file of 25.2MB full of 1's

But the python process uses 388MB of RAM.

Can anyone explain why it is using so much memory and maybe suggest a more memory efficient code?
** Python Support Thread ** Quote
11-13-2011 , 07:11 PM
immediate thought is just change range to xrange (assuming your on python 2.X)
** Python Support Thread ** Quote
11-13-2011 , 07:19 PM
Quote:
Originally Posted by Neko
immediate thought is just change range to xrange (assuming your on python 2.X)
2.7.2

Now it uses 1.7 MB of RAM.

Thanks
** Python Support Thread ** Quote
11-13-2011 , 07:31 PM
The reason it used so much ram before is because range() creates the entire object in memory before iterating over it, while xrange uses a generator expression (http://docs.python.org/library/functions.html#xrange).
** Python Support Thread ** Quote
11-13-2011 , 08:24 PM
I'm a newb trying to figure out the string format method for python 3. This is from http://learnpythonthehardway.org/book/ex24.html written for python 2.

Code:
def secret_formula(started):
    jelly_beans = started * 500
    jars = jelly_beans / 1000
    crates = jars / 100
    return jelly_beans, jars, crates

print "We'd have %d beans, %d jars, and %d crates." % secret_formula(start_point)
below doesn't work, giving "IndexError: tuple index out of range":

Code:
 print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))
but this does,

Code:
a,b,c = secret_formula(start_point)
print ("We'd have {0} beans, {1} jars, and {2} crates.".format(a,b,c))
what gives?
** Python Support Thread ** Quote
11-13-2011 , 08:37 PM
secret_formula(start_point) is a tuple of three items (at least I assume it is). So you can use it either as one item or as three. But Python can't always tell which you mean.

When you try
Code:
print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))
it interprets it as just one argument, which is the default.

When you do
Code:
a,b,c = secret_formula(start_point)
it is smart enough to realise that the tuple needs to be unpacked into three items which are then assigned to a, b and c.

To force it to unpack the tuple, you can use the * operator, like so:
Code:
print ("We'd have {0} beans, {1} jars, and {2} crates.".format(*secret_formula(start_point)))

Last edited by RoundTower; 11-13-2011 at 08:39 PM. Reason: not sure if this is the way the book expects you to do it, but it's the closest working way to the code you have
** Python Support Thread ** Quote
11-13-2011 , 08:58 PM
merci

Last edited by e i pi; 11-13-2011 at 08:58 PM. Reason: beaucoup
** Python Support Thread ** Quote
11-15-2011 , 02:15 AM
Optimization help please!

The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.

In the example below, we read each byte directly from the file into <byte_wiki>. This method takes approx 5-7 minutes and uses 1.8MB of RAM.

The commented out section reads the file before hand into a list and uses 690MB of ram and takes much longer to run, 9++ minutes.

nBytes = 25 million+

Code:
for b in xrange(1,nBytes+1):
    byte_copy = bin(random.randint(0,255))[2:].rjust(8,'0')
    #byte_wiki = bin(ord(source_wiki[b]))[2:].rjust(8,'0')
    byte_wiki = bin(ord(wiki7z.read(1)))[2:].rjust(8,'0')
    
    list_copy = []
    list_wiki = []
    for bit in byte_copy:
        list_copy.append(bit)
    for bit in byte_wiki:
        list_wiki.append(bit)
    for compare in xrange(8):
        if list_copy[compare] == list_wiki[compare]:
            score += 1
Why is this code so slow?

I am guessing that when the code is read from file that i/o is a bottleneck.
I am guessing that when fed into a list that the 25+million elements are what is slowing it down.

My guess is that splitting the Hex byte values and formatting them into lists, and then comparing each bit are expensive operations.

Also having list_copy = [] and list_wiki = [] within a loop looks silly to me but I couldn't think of another way around that.

Looks like I should probably learn how to do some bitwise manipulation tomorrow.
** Python Support Thread ** Quote
11-15-2011 , 02:44 AM
I'm rusty and not 100% that this is what you're trying to do, but it looks like you can dispense with all this list stuff and just XOR them...

Code:
for b in xrange(1,nBytes+1):
    byte_copy = random.randint(0,255)
    byte_wiki = ord(wiki7z.read(1))
    
    score += bin(byte_copy ^ byte_wiki).rjust(9, '0').count('0') - 1
EDIT: The "-1" is because I think it counts the '0b' at the beginning of the string form of the binary number and you don't want that. And I had to add back in the leading 0 padding. This will put the leading zeros in the wrong order but I don't think it matters since you will never use this string again from the looks of it.

EDIT2: Actually now that I think about it some more I think this works even better:

Code:
score += 8 - bin(byte_copy ^ byte_wiki).count('1')
Again, I'm rusty and actually just muddling through this for practice

Last edited by Xhad; 11-15-2011 at 03:03 AM.
** Python Support Thread ** Quote
11-15-2011 , 07:58 AM
You can use izip to iterate over the two lists at the same time. This way you don't need to do the copy loop followed by the compare loop.
** Python Support Thread ** Quote
11-15-2011 , 11:53 AM
what are you actually trying to do?

Quote:
The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.
i have a hard time thinking of why you would need to do this rather than just use 'cmp' to tell you if the files are the same or not. i guarantee 'cmp' will be faster than any python program you could ever write (for all but the most degenerate cases).
** Python Support Thread ** Quote
11-16-2011 , 01:45 AM
Code:
b = [1,2,3,4,5,6,7]
for g in b:
    print(b.pop(0))
this only returns 1-4 when run. why?
** Python Support Thread ** Quote
11-16-2011 , 01:54 AM
Quote:
Originally Posted by e i pi
Code:
b = [1,2,3,4,5,6,7]
for g in b:
    print(b.pop(0))
this only returns 1-4 when run. why?
I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.

Try:
Code:
for g in b:
    print g
or:
Code:
while b:
    print b.pop(0)
** Python Support Thread ** Quote
11-16-2011 , 02:35 AM
Quote:
Originally Posted by calm
I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.
That makes perfect sense, thanks.

Just playing around with this, python returns about half the list no matter the size and raises no errors.
** Python Support Thread ** Quote
11-16-2011 , 03:03 AM
oddly enough, if you make it "for i in range(len(b))" it works properly. I guess the "for x in b" construct makes it so that when you pop and iterate, whatever counting method it's using ends up moving two spots instead of one.
** Python Support Thread ** Quote
11-26-2011 , 08:47 PM
Using Python 3.

I can't figure out why r returns as None:

This is one of those things where I know I should know the answer, but for some reason....

Code:
def fac(t, r = 1):
    r *= t[-1]
    if len(t) == 1:
        return r
    fac(t[:-1], r)

def fun20():
    t = []
    for i in range(1, 101):
        t.append(i)
    a = fac(t)
    b = sum(list(map(int, str(a))))
    print(b)
breaks

while this will print out the answer:

Code:
def fac(t = t, r = 1):
    r *= t[-1]
    if len(t) == 1:
        print(sum(list(map(int, str(r)))))
    fac(t[:-1], r)
I'll torture you guys to add that I know this could be done in one line of code, but I decided to do this one the hard way because I wanted to be sure I can do a factorial without busting, which apparently, I can't.

Last edited by daveT; 11-26-2011 at 09:04 PM.
** Python Support Thread ** Quote
11-26-2011 , 09:51 PM
You forgot to put "return" at the end
** Python Support Thread ** Quote
11-26-2011 , 10:04 PM
<headslap.gif>
** Python Support Thread ** Quote
11-26-2011 , 10:53 PM
if it makes you feel any better I literally just made the same mistake
** Python Support Thread ** Quote
11-28-2011 , 12:14 PM
So I've been casually chugging along learn python the hard way until this,

http://learnpythonthehardway.org/book/ex46.html

where the author basically says download a bunch of testing and installation packages and figure out how they work on your own wtf! all the documentation is for linux also but I'm using windows.

I finally figured out how to install the packages but I can't get this nosetests thing to work. I've found this file, C:\python32\scripts\nosetests-script.py, that runs but it doesn't actually test anything. Do I have to pass in the directory of my "project skeleton" as an argument or something?

Also is any of this stuff necessary? or is there an easier way to do testing that doesn't require using the command prompt like its 1996? I would gladly give up on trying to figure this out at this point if this stuff is out dated or just unnecessary.
** Python Support Thread ** Quote
11-28-2011 , 09:11 PM
Normally you just have to do:

Code:
C:\Path\To\Your\Project> nosetests.exe .
Nose is an excellent testing framework. Your statement about command lines being out of date is misguided fwiw. The default windows shell is terribad but understanding how to use a command prompt will be a major productivity booster.

Last edited by Neko; 11-28-2011 at 09:29 PM.
** Python Support Thread ** Quote
** Python Support Thread **
150% up to $2,000 Welcome Bonus on CoinPoker
Join the action now
Daily Rewards • Splash Pots • CoinRaces
** Python Support Thread **

      
m