Maybe it's cause I'm new and don't understand everything well, but I'm starting to doubt whether it is justified to set mine with 88 and below. The reason for this is that I find that quite a bit of the time you hit a set, someone else ends up hitting a straight or flush and you lose quite a bit. Now obviously you can hit a boat, which does happen, but I don't know if that covers it.

Assuming we are paying all pots at 3BB to see the flop, I think it would be profitable to set mine if we won the pot 90%+ of the time we hit our set. Obviously I'm not delving into the math indepth here, but it feels like that is only around 65%. But given that you lose your 3BB the majority of the time you set mine, plus you lose significantly more (20BB+) the times you flop a set and someone else runs into higher value hand, how is it justified? What am I missing here?

Pure set mining is a losing play. Just not enough bad players to pay you off for all the times you miss. The hands you describe sound like coolers and selective memory bias.

Ok so I'm not insane...

Yeah probably some recency bias, but I just can't see how it is +EV to always set mine.

Well back in 2008, you could just call pre, fold if you missed your set and get stacks in if you hit. Nowadays, players are far more clued up and won't pay you off. You will sometimes get outdrawn by a better hand, though that's rare and usually just a cooler.

88 is playable as a call almost anywhere. 77 plays OK in late position or the blinds. 66-22 are more problematic and you won't be making a big mistake if you fold them in every position except for the blinds (especially the BB; always defend with pairs when it only costs you 2bb). 55-22 rarely get to showdown, so you only want to go set-mining with them if you're almost guaranteed to see a flop HU and it's cheap. The old idea that pairs play best multiway is no longer true, online at least. Decent players rarely play for stacks multiway with anything that loses to bottom set, so a lot of "old skool" players actually find they get destroyed with small pairs, as they run into oversets/straights/flushes when the pots get huge.
There's a ton of variance with set-mining though, due to the low frequency of getting dealt a pair and then actually spiking the set, so your tracked results can be all over the place for tens of thousands of hands. (Some people run like god with sets. Some get crushed. You can reduce your variance without giving up much theoretical EV by folding the smallest pairs).