Say, "Well, have you donated all of your money to solve world hunger yet?" Everyone will become a bit sheepish. Then say, "But it's a nice gesture!...Yeah, I thought so."
So, I think I'm pretty polite about the whole thing despite the constant pressure and once I've explained myslef, I say "I hope there's no hard feelings." And in return I get "Yeah there's hard feelings - I think it's very selfish."