lots of time to do work today. goal is to make program to go through oddsportal.com or sbr and get the pinny line for each NFL game over last few years and also the scores

how do you know if you dont have experience with either of those?

lol okay i have made a program succesfully parsing HTML with regex. although admittedly that was because i was too n00bie to understand those libraries. i will take your advice though and go about it as you say...

The point is that regex is desigened to to work with a specific class of text and HTML is not that type. As stated by other answers in that thread, regex can be used when you are trying to parse a very speceific sub-set of HTML. However, if you are trying to do something more complex or do not have control of the input format, using a framework specifically designed for parsing HTML will make your life easier.

As for an actual recomendation, I've seen beautiful soup used and talked about frequently and never heard of elementtree. So based on that, I suspect it would be easier to get help with any potential issues if you use beautiful soup.

hey sorry if i ask too many questions. i have no formal instruction in programming and am learning this as i go along with no deep understanding of what is happening. after 2 hours of work i figured out how to get BeautifulSoup version 3 on my computer by downloading it, extracting the .tar.gz file and then run the setup file that it has which builds a file BeautifulSoup which i can cut and paste into the python/lib directory

however, i am trying to get BeautifulSoup version 4 on here and when i extract and run the setup file it doesn't build a BeautifulSoup file which i can use. here is the documentation which i don't understand. i don't know where to run these commands or what they do or what pip is. i downloaded easy install but still can't get this to work.

Your process should roughly go like this:

You should never have to manually copy the files anywhere.

Alternatively install pip: http://www.saltycrane.com/blog/2010/...ll-pip-ubuntu/

and then from the command line:

Thanks Neko. I'm going to need to install some packages soon and that info will be helpful.

i haven't used beautiful soup but people at work like it.

we mostly use pyquery at work. it has been adequate for my (simple) needs.

beautifulsoup is so much better than elementtree, with elementtree can't even access a tag's siblings so gay

preventing incest is gay? these kids today...

how does this work when i tried to setup lxml myself i couldn't, how does this program work?

http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

I wrote a program in python which runs math-intensive simulations. I usually run 7,500 simulations when I run this code.

The problem is it runs slow and I want to improve it. I spent a good amount of time setting up a threading process, where I had 50-75 simulations running at once. However I discovered that this didn't speed up my program at all. It ran just as a fast using a loop where I did 1 simulation per loop.

Is this because the CPU only has a limited amount of processors (1?) capable of running math calculations, so the threads just end up waiting on each other (effectively mimicking a loop where 1 simulation happens per iteration)?

This answers my question I believe: http://stackoverflow.com/questions/3...and-performace

yeah...definitely the GIL.

Here are some options for speeding up your calculations


- If your simulations aren't dependent on each other, just run multiple python interpreters.

- Use numpy especially if you're working with matricies

- Use the multiprocessing module instead of threading. This will allow you to run multiple simulations in parallel.

- Try Scipy.weave

- Rewrite any hotspots in your code using Cython. You can very easily get some big gains (like orders of magnitude better in some cases)

- Write performance critical parts in C and make them accessible via a dll and then use ctypes to call them.

- Write a full fledged C-extension

before you try any of this, the first step imo is to identify the bottlenecks. There may be 20 different parts of your code that could be sped up tenfold, but only one or two that will actually give a significant increase in your overall running time.

Start with the profile or cProfile modules in the standard library.

yeah that's a good point to make. Profiling (use cProfile rather than profile) your program to find out where it's spending most of its time is the obvious first step.

here is a programming problem i am having a problem with. i have data for 1300 football games. for each game i have the spread, and what the result was compared to the spread. it is set up as a list of lists now [(game1_spread, game1_result_compared_to_spread), (game2_spread, game2_result_compared_to_spread), etc.]

i want to take this information and make a printout with these two pieces of data:

spread: game1_result_compared_to_spread==0/total_games_with_this_spread
spread2: game1_result_compared_to_spread==0/total_games_with_this_spread
etc. for each spread

how can i get this?

okay did it i will post solution and appreciate feedback

Brand new to programming, just started going through the MIT opencourse and have some questions regarding Python.

If I open "Python Shell" and from there open "New Window" is this where I will be writing my programs? I then save it there and use the F5 function to run it in the shell correct?

Why when I am using that "New Window" does it automatically indent sentences that I don't think it should? For example I type the following for the first line:

outstanding_balance = int(raw_input("What is your outstanding balance?")

When I press enter it indents me about 7 tab spaces on the next line like:

outstanding_balance = int(raw_input("What is your outstanding balance?")
print "What is your outstanding balance?"


also if I'm not mistaken my first command is saying that whatever the user inputs when prompted, it will be turned from a string into an integer correct? And the second command should print the prompt for the user, correct?

When I try to run that code I get a syntax error.

Edit to add: 2+2 isn't displaying the indent the way I'm intending to show it, imagine "print" starting directly under "raw_input"

Ok, for starters I'm not quite sure what you mean by "Python shell" and "New Window", but I can answer your other questions.

As for this line:
The indent is happening because your parenthesis aren't balanced. You need one ")" to close the "raw_input(" function, and another ")" to close the "int(" function.

Yes you are correct. The int() function will convert an object (in this case a String) into an integer. Hint: Type "help(int())" in your python shell.

The reason you are getting an error could be for a few reasons. As I said before you need to close your open parenthesis. In addition, the int() function will only work on a properly formatted String.

Try the following:
int('1')
int('3.4')
int('two')

That reminded me of this exchange for some reason...

hahahahha

MUCH LOVE FROM ME.

Yea im doing this course too and thought the compiler they use seems kinda crappy unless im missing something.

i do the same as you but i dont know how to run the debugger from a new window and a few other things.

god i hate elementtree and namespaces

heh, and you're just scraping it. imaging trying to write it!

http://odds.sbrfeeds.com/events/?new...d&version=1.0&

can i get some help here, i have this page. how can i access the elements <side-name> using Xpath? how can i get access the element <side-name> where it's text equals "Jacksonville"