You have the chip leader on your left but you can't just keep folding for that reason.
To an extent he also has to give you credit for having a hand when you are risking your tournament life.
If he called you with a pair or ace then it's just variance. If he called you with king rag then make a note if this a player pool in which you are going to see him a lot.