Tough decision but not a lot of $ at risk. I'd call. Fold is reasonable though.
IMO you showed weakness post flop and figured J
was scare card. When you called, he figured you were still scared. Pure speculation. For all we know, he could've flopped trips.
Your play wasn't conservative but it was passive, and you might have put yourself in a position to get pushed around by a bigger stack.
If you're in this type of situation again, try betting the flop after everybody else checks (size $30-$40) or raising Villain on the turn once you picked up some outs (aim for 2.5-3x his bet). Try either one but not both. Might seem risky but you may get some surprising results.
How'd this hand turn out?