TLDR, except you may be considering going to grad school if you are reading this thread, in this case there are no more TLDR's for you.
Just reading this thread and being a grad school dropout, I would like to take the opportunity to discourage as many people as possible from going to grad school. I think it is a huge beat for about 70% of the people who go, the best thing possible for maybe 10%, and EV neutral or thereabouts for the rest. Most of the reasons for this are clearly laid out ITT. Basically for there to be any chance of all of going to grad school to be a good decision for you, you should have a story similar to the one Max Raker posted here. Of the many people I knew/know, those were the guys that enjoyed grad school, although not all did well. The normal people, who just liked the subject and wanted to learn more, wanted a challenge, thought it sounded cool, etc., it is not for them.
For all the glory of doing original research on an exciting topic you like, it really boils down to working 60+ hrs a week for 20 grand a year. Even in the poor economy these days (which is not as bad as they say for people with a shot of getting into a good grad school) that's not a good deal unless you come from China or India, which is where 40% of your classmates will be from, and even many of them leave to get jobs after two years (they mostly just want the student visa, maybe this was just a physics thing though). Some may say that they are not interested in money, my personal experience is that this changes for most people after they get a taste of the moneyed lifestyle a real job can bring. Also, some people may want to start a family (this was a big reason for me leaving). Good luck doing this when your first paying long-term job won't come until you are 28.
About the research, what you will often find is that much of what is being done is not done so much to be cutting edge or really make big advancements as it is to satisfy those in charge of issuing the funding to get the next big grant. Almost regardless of field, it is highly likely that a large portion of your time will be spent on work more related to this issue than your dissertation. At least in physics but I know of this to be true in many types of engineering as well, most of the work being done in experiment these days involves either highly incremental advancements, or proving stuff that is basically overwhelmingly likely to be true, since the risks are too high on both PI and funding manager sides for more interesting projects. I found this to be one of the extremely depressing aspects of academia in general.
The opportunity costs are really pretty similar to poker - most of the people capable of succeeding in grad school in something like the hard sciences would also be capable of doing extremely well in a wide variety of careers. The level of intelligence and especially work ethic of the people you meet at any top tier institution in a hard science is going to be incredibly high. This is why you see Ph.D.'s filling so many of the quant and consulting jobs out there - these people couldn't care less about your wanker particle physics or whatever you did, they just want someone willing and able to work long hours with minimal help while often being treated like dirt, which is what the Ph.D. is. Now, you may very well be incredibly intelligent with a very strong work ethic. The question is, do you want to enter something where basically everyone else also meets these criteria, and the rewards are mediocre at best, when you could go into business or engineering and probably soul crush people?
About admissions, grades are important, get at least a 3.7 but 3.5+ have a chance, if there is a subject GRE in your field, that is very important and the regular GRE is almost useless. If there is not a subject GRE, the regular may well be important but it is a very poor differentiator as most people will get close to 800. How it really works though is that most of the students that are actually admitted either have profiles that are far, far above the average profile for the class, or profiles somewhere near the average combined with strong letter(s) of recommendation from a professor that is either extremely well-known in the field or has a personal relationship with someone on the committee. Publications are not even worth worrying about unless you get extremely lucky to be a first author on a major paper in a top journal, basically unheard of, and the content of the letters is not even that important other than it be positive and related to research. The single most important thing is that at least one of them comes from someone known by a member of the committee. Remember also that at the top schools, everyone has a near-perfect profile, so there is little room to get in solely on the basis of strong academics. By the way, many schools will post an average profile of accepted candidates. Remember that 40% of the candidates come from China and India. These students nearly always have perfect GPA's and GRE scores (other than verbal) and there are tons and tons of applicants coming from these countries, most of them borrow the money and send apps to 20+ schools, often paying to have their personal statement and recommendations written for them. As an American student, for various reasons, you can get in with slightly lower qualifications and your odds are not as dismal as what will be listed although I again stress the importance of the letters of recommendation.
As for what school to go to, I would suggest you are wasting your time if you want a career in academics but aren't going to one of the top 20 schools, and again, the very biggest names are much better. And if you want to get a Ph.D. followed by a job, you are 100% making a horrible decision anyway, go get a job now and then have a better one with your seven years of experience in seven years. If you look at the names of tenure track researchers, even if you are willing to go as low as the top 50 schools, nearly all got their Ph.D. from schools in the top 10-15 in the field, or the top 1-3 in their country of origin. Some of this is due to the selection bias that the top students in undergrad naturally get into the best schools and so they will of course be the ones most likely to succeed in research and go on to get the best jobs, but honestly this probably only accounts for about 30-40% of this. The fact is that in academics, all that really matters is name recognition, and people DO automatically assume that you are more competent if you went to a top school and the opposite if you went to Podunk U., and this even includes people at institutions outside of this category. In fact even with an awesome pub record you will still face major challenges moving into more prestigious institutions from grad to postdoc.
Finally, some advice I wish I had, I think many rip on people in the academic fields for having poor social skills, and there will be tons of these people around that you will encounter in grad school. Let me assure you that social skills are just as important for grad students and maybe more so for anyone above that in academia as they are in an actual workplace. Most of the stuff you will be working on, should you choose to go to grad school (and you probably should not) is going to be too hard to figure out on your own, especially if it is worth doing. In both academic and business worlds the key to getting a project done is finding the right person or people that know how to accomplish a subset of your project and convincing them to do it for you (actually you get them to tell you how to do it since grad students do all the actual work), until you can get to the subset that you are capable of doing yourself. This subset may well be zero, despite this you may well have made the most significant contribution simply in getting everyone else's knowledge together in a useful way. In any case, if you are lucky this person you need to talk to will be your advisor or someone in your group, if you are less lucky a collaborator, below that a random person such as another professor or engineer (but it is still very fortunate that you even know this person), and worst of all but also most common, a competitor or other group that will never quite tell you what you need to know. In either case it is extremely important to your research that you communicate well with these busy people and convince them that you are worth their time. This will come not only in talking with them directly, but also in your presentation of scientific talks and other representations of you and your research group - and I hope you enjoy giving these talks because you will be giving many even as a grad student often to the detriment of more useful things.
The good part about going and getting a Ph.D. in the sciences is that it's usually not that bad if you don't like it and decide to quit. You can usually get a masters after like a year and a half whether you did well or not, and that is worth something. The key is to try and decide early on if you are going to stay, then if not (and even if you are really) try and put yourself in a position through research where you can get some viable, employable skills. All fields have this and most but not all research projects will involve useful skills. Furthermore if you go to a good school you'll probably have a nice alumni network and career center that can land you some interviews, plus you may be able to leverage your undergrad as well, from there it is basically up to you to get hired. Just don't expect to get anything special with a masters, you will likely be applying to the same jobs as undergrads unless you have some extremely relevant work experience. I personally got a job that I could have got as an undergrad, only it would have been very unlikely for me to actually get that job when I was an undergrad, this is the case for other people I knew in my position as well.