I couldn't tell you how young I started. Like you, I grew up in it. My mother's family played cribbage and drank gin & tonics. I picked up one, but not the other.
Every house had 2 to 5 boards in it. I don't recall muggins ever being a factor; not that the rule was not in effect, it just wasn't necessary.
My grandmother also played bridge and golfed. I never got into playing golf (but I enjoy watching it), and I took bridge lessons when I was 13. I played socially until my kids were born (long time ago), and I want to get back into that. For a while, my mother played a weekly game with her friends.
I've always been into card games. When I was younger I used to have a deck on me at all times. Sometimes I would just shuffle to pass the time.
/ramble