I had this discussion once when I was taking a psych course as an undergrad (many years ago). I had the opinion prevalent here, sight seems more important in general and I assumed I would miss it the most. The professor got involved and disagreed. He claimed that studies showed that loss of hearing caused greater emotional problems due to the severe impairment of human communication.
I think I would like to hold on to both, but the answer may be more complicated than this thread has recognized. I did a quick search and got this which seems to agree with the instinctive opinion:
http://www.ncbi.nlm.nih.gov/pubmed/8356208
Just food for thought.