** Python Support Thread ** - Page 13 - Computer Technical Help

Two Plus Two Forums Other Topics Computer and Technical Help

** Python Support Thread **

Post Reply Subscribe

...

Page 13 of 30

First

3 8 9 10 11 12 13 14 15 16 17 18 23

Last

Page 13 of 30

First

3 8 9 10 11 12 13 14 15 16 17 18 23

Last

11-13-2011 , 05:17 PM

#301

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Quote:

Originally Posted by calm

Yes I want option 2, but as a list comprehension instead of a for loop. Think I got it figured out thanks to Neko.

Here's another for loop I'm trying to substitute with a list comprehension:

Code:

row_data = []
row_color = []
for key in dict:
    row_data.append([])
    row_color.append([])

Now I could easily do this with two separate list comprehensions:

Code:

row_data = [[] for key in dict]
row_color = [[] for key in dict]

But since I'm then looping over the dict twice, I guess the for loop would be faster? Is there a way to combine the two in one list comprehension?

would definitely prefer the 2 line solution over 5 lines. You can use the timeit module to figure out which way is faster

Code:

import timeit

d = dict(zip(range(100),range(100)))

def method1():
    row_data = []
    row_color = []
    for key in d:
        row_data.append([])
        row_color.append([])
        
        
def method2():
    row_data = [[] for key in d]
    row_color = [[] for key in d]

t1 = timeit.Timer("method1()","from __main__ import method1,d")
t2 = timeit.Timer("method2()","from __main__ import method2,d")

print "t1"
print t1.timeit(10000)
print "t2"
print t2.timeit(10000)

output:

Code:

t1
2.03231943169
t2
0.910326269734

Quote

150% up to $2,000 Welcome Bonus on CoinPoker

Join the action now

Daily Rewards • Splash Pots • CoinRaces

11-13-2011 , 05:49 PM

#302

calm

banned

Join Date: Jul 2006 Posts: 2,087

Cool thanks. Haha I actually tried using timeit but got stuck an infinite loop somehow.

Last edited by calm; 11-13-2011 at 05:52 PM. Reason: Oh I see it defaults to one million trials lol, that was the problem

Quote

11-13-2011 , 06:26 PM

#303

RoundTower

ɹǝʍoʇpunoɹ

Join Date: Feb 2005 Posts: 14,562

If you only want to do the dict comprehension once, you can do

Code:

row_data, row_color = zip(*(([],[]) for key in dic))

or if it's vitally important that row_data and row_color end up being lists instead of tuples,

Code:

row_data, row_color = map(list, zip(*(([],[]) for key in dic)))

If you think this is unreadable then just write it the other way, but it's good to know you have this option in case you do end up with this code in a performance bottleneck. zip(*spam) is a very useful idiom for transposing lists.

Quote

11-13-2011 , 06:55 PM

#304

RoundTower

ɹǝʍoʇpunoɹ

Join Date: Feb 2005 Posts: 14,562

of course, if all you want is two lists of empty lists, each the same length as the dictionary, there are lots of efficient ways to do it. But the above is the generalizable version.

Quote

11-13-2011 , 07:02 PM

#305

stanek

grinder

Join Date: Apr 2006 Posts: 452

Code:

file = open("test.bin","wb")

for b in range(25209495):
    file.write("\xFF")

This writes a file of 25.2MB full of 1's

But the python process uses 388MB of RAM.

Can anyone explain why it is using so much memory and maybe suggest a more memory efficient code?

Quote

11-13-2011 , 07:11 PM

#306

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

immediate thought is just change range to xrange (assuming your on python 2.X)

Quote

11-13-2011 , 07:19 PM

#307

stanek

grinder

Join Date: Apr 2006 Posts: 452

Quote:

Originally Posted by Neko

immediate thought is just change range to xrange (assuming your on python 2.X)

2.7.2

Now it uses 1.7 MB of RAM.

Thanks

Quote

11-13-2011 , 07:31 PM

#308

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

The reason it used so much ram before is because range() creates the entire object in memory before iterating over it, while xrange uses a generator expression (http://docs.python.org/library/functions.html#xrange).

Quote

11-13-2011 , 08:24 PM

#309

e i pi

veteran

Join Date: Feb 2010 Posts: 2,230

I'm a newb trying to figure out the string format method for python 3. This is from http://learnpythonthehardway.org/book/ex24.html written for python 2.

Code:

def secret_formula(started):
    jelly_beans = started * 500
    jars = jelly_beans / 1000
    crates = jars / 100
    return jelly_beans, jars, crates

print "We'd have %d beans, %d jars, and %d crates." % secret_formula(start_point)

below doesn't work, giving "IndexError: tuple index out of range":

Code:

 print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))

but this does,

Code:

a,b,c = secret_formula(start_point)
print ("We'd have {0} beans, {1} jars, and {2} crates.".format(a,b,c))

what gives?

Quote

11-13-2011 , 08:37 PM

#310

RoundTower

ɹǝʍoʇpunoɹ

Join Date: Feb 2005 Posts: 14,562

secret_formula(start_point) is a tuple of three items (at least I assume it is). So you can use it either as one item or as three. But Python can't always tell which you mean.

When you try

Code:

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(secret_formula(start_point)))

it interprets it as just one argument, which is the default.

When you do

Code:

a,b,c = secret_formula(start_point)

it is smart enough to realise that the tuple needs to be unpacked into three items which are then assigned to a, b and c.

To force it to unpack the tuple, you can use the * operator, like so:

Code:

print ("We'd have {0} beans, {1} jars, and {2} crates.".format(*secret_formula(start_point)))

Last edited by RoundTower; 11-13-2011 at 08:39 PM. Reason: not sure if this is the way the book expects you to do it, but it's the closest working way to the code you have

Quote

11-13-2011 , 08:58 PM

#311

e i pi

veteran

Join Date: Feb 2010 Posts: 2,230

merci

Last edited by e i pi; 11-13-2011 at 08:58 PM. Reason: beaucoup

Quote

11-15-2011 , 02:15 AM

#312

stanek

grinder

Join Date: Apr 2006 Posts: 452

Optimization help please!

The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.

In the example below, we read each byte directly from the file into <byte_wiki>. This method takes approx 5-7 minutes and uses 1.8MB of RAM.

The commented out section reads the file before hand into a list and uses 690MB of ram and takes much longer to run, 9++ minutes.

nBytes = 25 million+

Code:

for b in xrange(1,nBytes+1):
    byte_copy = bin(random.randint(0,255))[2:].rjust(8,'0')
    #byte_wiki = bin(ord(source_wiki[b]))[2:].rjust(8,'0')
    byte_wiki = bin(ord(wiki7z.read(1)))[2:].rjust(8,'0')
    
    list_copy = []
    list_wiki = []
    for bit in byte_copy:
        list_copy.append(bit)
    for bit in byte_wiki:
        list_wiki.append(bit)
    for compare in xrange(8):
        if list_copy[compare] == list_wiki[compare]:
            score += 1

Why is this code so slow?

I am guessing that when the code is read from file that i/o is a bottleneck.
I am guessing that when fed into a list that the 25+million elements are what is slowing it down.

My guess is that splitting the Hex byte values and formatting them into lists, and then comparing each bit are expensive operations.

Also having list_copy = [] and list_wiki = [] within a loop looks silly to me but I couldn't think of another way around that.

Looks like I should probably learn how to do some bitwise manipulation tomorrow.

Quote

11-15-2011 , 02:44 AM

#313

Xhad

Carpal \'Tunnel

Join Date: Jul 2005 Posts: 10,962

I'm rusty and not 100% that this is what you're trying to do, but it looks like you can dispense with all this list stuff and just XOR them...

Code:

for b in xrange(1,nBytes+1):
    byte_copy = random.randint(0,255)
    byte_wiki = ord(wiki7z.read(1))
    
    score += bin(byte_copy ^ byte_wiki).rjust(9, '0').count('0') - 1

EDIT: The "-1" is because I think it counts the '0b' at the beginning of the string form of the binary number and you don't want that. And I had to add back in the leading 0 padding. This will put the leading zeros in the wrong order but I don't think it matters since you will never use this string again from the looks of it.

EDIT2: Actually now that I think about it some more I think this works even better:

Code:

score += 8 - bin(byte_copy ^ byte_wiki).count('1')

Again, I'm rusty and actually just muddling through this for practice

Last edited by Xhad; 11-15-2011 at 03:03 AM.

Quote

11-15-2011 , 07:58 AM

#314

TheIrishThug

Pooh-Bah

Join Date: Jan 2005 Posts: 5,223

You can use izip to iterate over the two lists at the same time. This way you don't need to do the copy loop followed by the compare loop.

Quote

11-15-2011 , 11:53 AM

#315

tyler_cracker

Carpal \'Tunnel

Join Date: Apr 2005 Posts: 15,828

what are you actually trying to do?

Quote:

The idea of this program is that we have a binary file that is 25.2MB big and we want to compare it bit by bit and count the number of bits that match.

i have a hard time thinking of why you would need to do this rather than just use 'cmp' to tell you if the files are the same or not. i guarantee 'cmp' will be faster than any python program you could ever write (for all but the most degenerate cases).

Quote

11-16-2011 , 01:45 AM

#316

e i pi

veteran

Join Date: Feb 2010 Posts: 2,230

Code:

b = [1,2,3,4,5,6,7]
for g in b:
    print(b.pop(0))

this only returns 1-4 when run. why?

Quote

11-16-2011 , 01:54 AM

#317

calm

banned

Join Date: Jul 2006 Posts: 2,087

Quote:

Originally Posted by e i pi

Code:

b = [1,2,3,4,5,6,7]
for g in b:
    print(b.pop(0))

this only returns 1-4 when run. why?

I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.

Try:

Code:

for g in b:
    print g

or:

Code:

while b:
    print b.pop(0)

Quote

11-16-2011 , 02:35 AM

#318

e i pi

veteran

Join Date: Feb 2010 Posts: 2,230

Quote:

Originally Posted by calm

I'm surprised that code even works - you don't want to loop through the list while "popping" items at the same time.

That makes perfect sense, thanks.

Just playing around with this, python returns about half the list no matter the size and raises no errors.

Quote

11-16-2011 , 03:03 AM

#319

Xhad

Carpal \'Tunnel

Join Date: Jul 2005 Posts: 10,962

oddly enough, if you make it "for i in range(len(b))" it works properly. I guess the "for x in b" construct makes it so that when you pop and iterate, whatever counting method it's using ends up moving two spots instead of one.

Quote

11-26-2011 , 08:47 PM

#320

daveT

S.A.G.E. Master

Join Date: Jun 2005 Posts: 23,955

Using Python 3.

I can't figure out why r returns as None:

This is one of those things where I know I should know the answer, but for some reason....

Code:

def fac(t, r = 1):
    r *= t[-1]
    if len(t) == 1:
        return r
    fac(t[:-1], r)

def fun20():
    t = []
    for i in range(1, 101):
        t.append(i)
    a = fac(t)
    b = sum(list(map(int, str(a))))
    print(b)

breaks

while this will print out the answer:

Code:

def fac(t = t, r = 1):
    r *= t[-1]
    if len(t) == 1:
        print(sum(list(map(int, str(r)))))
    fac(t[:-1], r)

I'll torture you guys to add that I know this could be done in one line of code, but I decided to do this one the hard way because I wanted to be sure I can do a factorial without busting, which apparently, I can't.

Last edited by daveT; 11-26-2011 at 09:04 PM.

Quote

11-26-2011 , 09:51 PM

#321

Xhad

Carpal \'Tunnel

Join Date: Jul 2005 Posts: 10,962

You forgot to put "return" at the end

Quote

11-26-2011 , 10:04 PM

#322

daveT

S.A.G.E. Master

Join Date: Jun 2005 Posts: 23,955

<headslap.gif>

Quote

11-26-2011 , 10:53 PM

#323

Xhad

Carpal \'Tunnel

Join Date: Jul 2005 Posts: 10,962

if it makes you feel any better I literally just made the same mistake

Quote

11-28-2011 , 12:14 PM

#324

e i pi

veteran

Join Date: Feb 2010 Posts: 2,230

So I've been casually chugging along learn python the hard way until this,

http://learnpythonthehardway.org/book/ex46.html

where the author basically says download a bunch of testing and installation packages and figure out how they work on your own

wtf! all the documentation is for linux also but I'm using windows.

I finally figured out how to install the packages but I can't get this nosetests thing to work. I've found this file, C:\python32\scripts\nosetests-script.py, that runs but it doesn't actually test anything. Do I have to pass in the directory of my "project skeleton" as an argument or something?

Also is any of this stuff necessary? or is there an easier way to do testing that doesn't require using the command prompt like its 1996? I would gladly give up on trying to figure this out at this point if this stuff is out dated or just unnecessary.

Quote

11-28-2011 , 09:11 PM

#325

Neko

Pooh-Bah

Join Date: Jul 2005 Posts: 4,000

Normally you just have to do:

Code:

C:\Path\To\Your\Project> nosetests.exe .

Nose is an excellent testing framework. Your statement about command lines being out of date is misguided fwiw. The default windows shell is terribad but understanding how to use a command prompt will be a major productivity booster.

Last edited by Neko; 11-28-2011 at 09:29 PM.